[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:21:56 dist-test-slave-zzwf krb5kdc[32693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:21:56 dist-test-slave-zzwf krb5kdc[32693](info): set up 2 sockets
Jun 20 20:21:56 dist-test-slave-zzwf krb5kdc[32693](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 20:21:57.738394 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:21:57 dist-test-slave-zzwf krb5kdc[32693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:21:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:21:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:21:57.913405 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:36105
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:35069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:36105
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:58.232896 32706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:58.233274 32706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:58.233459 32706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:58.240953 32706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:21:58.241120 32706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:58.241230 32706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:58.241330 32706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:21:58.241413 32706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:21:58.251807 32706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:35069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:36105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:36105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:58.253428 32706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:58.255390 32706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:58.268940 32711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:58.268908 32712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:58.269253 32714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:58.270247 32706 server_base.cc:1034] running on GCE node
I20260620 20:21:58.271129 32706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:58.272691 32706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:58.273927 32706 hybrid_clock.cc:648] HybridClock initialized: now 1781986918273883 us; error 46 us; skew 500 ppm
I20260620 20:21:58.274313 32706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:58 dist-test-slave-zzwf krb5kdc[32693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986918, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:58.277803 32706 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:21:58.279425 32706 webserver.cc:460] Webserver started at http://127.31.235.190:45693/ using document root <none> and password file <none>
I20260620 20:21:58.280184 32706 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:58.280341 32706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:58.280678 32706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:58.283998 32706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "0a80a62eede643fba4fe0ca17e8104e1"
format_stamp: "Formatted at 2026-06-20 20:21:58 on dist-test-slave-zzwf"
server_key: "b460db12e0b818edeeca56f7c5e680e2"
server_key_iv: "3dc05d14bdbf639fc20bab41f830de3e"
server_key_version: "encryptionkey@0"
I20260620 20:21:58.284844 32706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "0a80a62eede643fba4fe0ca17e8104e1"
format_stamp: "Formatted at 2026-06-20 20:21:58 on dist-test-slave-zzwf"
server_key: "b460db12e0b818edeeca56f7c5e680e2"
server_key_iv: "3dc05d14bdbf639fc20bab41f830de3e"
server_key_version: "encryptionkey@0"
I20260620 20:21:58.290539 32706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 20:21:58.294277 32721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:58.295528 32706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:21:58.295717 32706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "0a80a62eede643fba4fe0ca17e8104e1"
format_stamp: "Formatted at 2026-06-20 20:21:58 on dist-test-slave-zzwf"
server_key: "b460db12e0b818edeeca56f7c5e680e2"
server_key_iv: "3dc05d14bdbf639fc20bab41f830de3e"
server_key_version: "encryptionkey@0"
I20260620 20:21:58.295917 32706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:58.319489 32706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:58.323583 32706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:58.323882 32706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:58.342198 32706 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:36105
I20260620 20:21:58.342217 304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:36105 every 8 connection(s)
I20260620 20:21:58.343534 32706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:21:58.347095 305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:21:58.350767 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 32706
I20260620 20:21:58.351075 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:21:58.352916 32686 external_mini_cluster.cc:1428] Setting key 9e4af138ca9232c7c4e07cddefccaac8
I20260620 20:21:58.357214 305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1: Bootstrap starting.
I20260620 20:21:58.360379 305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1: Neither blocks nor log segments found. Creating new log.
I20260620 20:21:58.361359 305 log.cc:824] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1: Log is configured to *not* fsync() on all Append() calls
I20260620 20:21:58.364686 305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1: No bootstrap required, opened a new log
I20260620 20:21:58.375864 305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a80a62eede643fba4fe0ca17e8104e1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36105 } }
I20260620 20:21:58.376272 305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:21:58.376402 305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a80a62eede643fba4fe0ca17e8104e1, State: Initialized, Role: FOLLOWER
I20260620 20:21:58.377058 305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a80a62eede643fba4fe0ca17e8104e1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36105 } }
I20260620 20:21:58.377341 305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:21:58.377483 305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:21:58.377669 305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:21:58.380502 305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a80a62eede643fba4fe0ca17e8104e1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36105 } }
I20260620 20:21:58.381086 305 leader_election.cc:302] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a80a62eede643fba4fe0ca17e8104e1; no voters:
I20260620 20:21:58.381815 305 leader_election.cc:288] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:21:58.381914 310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:21:58.383157 310 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1 [term 1 LEADER]: Becoming Leader. State: Replica: 0a80a62eede643fba4fe0ca17e8104e1, State: Running, Role: LEADER
I20260620 20:21:58.383788 310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a80a62eede643fba4fe0ca17e8104e1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36105 } }
I20260620 20:21:58.383978 305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 20:21:58 dist-test-slave-zzwf krb5kdc[32693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:21:58.386545 312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a80a62eede643fba4fe0ca17e8104e1. Latest consensus state: current_term: 1 leader_uuid: "0a80a62eede643fba4fe0ca17e8104e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a80a62eede643fba4fe0ca17e8104e1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36105 } } }
I20260620 20:21:58.386909 312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:58.387482 311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a80a62eede643fba4fe0ca17e8104e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a80a62eede643fba4fe0ca17e8104e1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36105 } } }
I20260620 20:21:58.387770 311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:21:58.396656 316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:21:58.399399 307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:58.356787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49726 (local address 127.31.235.190:36105)
0620 20:21:58.357706 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:58.357723 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:58.359042 (+ 1319us) server_negotiation.cc:407] Connection header received
0620 20:21:58.370536 (+ 11494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:58.370561 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:58.370987 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:58.371357 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:58.375290 (+ 3933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:58.376867 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:58.379359 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:58.380019 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:58.380172 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:58.388293 (+ 8121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:58.388308 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:58.388331 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:58.388388 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:58.393300 (+ 4912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:58.394581 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:58.394593 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:58.394604 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:58.394876 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:58.395516 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:58.395525 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:58.395531 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:58.396365 (+ 834us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:58.396683 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:58.397864 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0620 20:21:58.398189 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":383,"threads_started":1}
I20260620 20:21:58.404415 316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:21:58.412389 316 catalog_manager.cc:1347] Generated new cluster ID: 5d6e3fea9dba4480a05bdb528a7814de
I20260620 20:21:58.412500 316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:21:58.438905 316 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:21:58.440240 316 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:21:58.451195 316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a80a62eede643fba4fe0ca17e8104e1: Generated new TSK 0
I20260620 20:21:58.452186 316 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
I20260620 20:21:58.503500 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36105
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:35069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:58.957203 332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:58.957561 332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:58.957765 332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:58.965261 332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:58.965379 332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:58.965611 332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:21:58.975966 332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:35069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36105
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:58.977363 332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:58.978793 332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:58.992606 337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:58.992749 340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:58.992666 338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:58.994063 332 server_base.cc:1034] running on GCE node
I20260620 20:21:58.994757 332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:58.995898 332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:58.997103 332 hybrid_clock.cc:648] HybridClock initialized: now 1781986918997043 us; error 50 us; skew 500 ppm
I20260620 20:21:58.997452 332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:58 dist-test-slave-zzwf krb5kdc[32693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986918, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:59.001008 332 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:21:59.002570 332 webserver.cc:460] Webserver started at http://127.31.235.129:43331/ using document root <none> and password file <none>
I20260620 20:21:59.003269 332 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:59.003373 332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:59.003629 332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:59.006657 332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "2c869cfb52484a1ba3b78e1e882c0c01"
format_stamp: "Formatted at 2026-06-20 20:21:58 on dist-test-slave-zzwf"
server_key: "f57340b0dcf22dda948d50c8d51a5018"
server_key_iv: "aac626c7119edd1bcf0e72880fbafaf8"
server_key_version: "encryptionkey@0"
I20260620 20:21:59.007409 332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "2c869cfb52484a1ba3b78e1e882c0c01"
format_stamp: "Formatted at 2026-06-20 20:21:58 on dist-test-slave-zzwf"
server_key: "f57340b0dcf22dda948d50c8d51a5018"
server_key_iv: "aac626c7119edd1bcf0e72880fbafaf8"
server_key_version: "encryptionkey@0"
I20260620 20:21:59.013036 332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 20:21:59.016903 347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:59.018520 332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:21:59.018746 332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "2c869cfb52484a1ba3b78e1e882c0c01"
format_stamp: "Formatted at 2026-06-20 20:21:58 on dist-test-slave-zzwf"
server_key: "f57340b0dcf22dda948d50c8d51a5018"
server_key_iv: "aac626c7119edd1bcf0e72880fbafaf8"
server_key_version: "encryptionkey@0"
I20260620 20:21:59.018989 332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:59.032356 332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:59.037484 332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:59.037892 332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:59.039151 332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:59.040678 332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:21:59.040784 332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:59.040951 332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:21:59.041028 332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:59.076061 332 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:38771
I20260620 20:21:59.076154 459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:38771 every 8 connection(s)
I20260620 20:21:59.077476 332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:21:59.084959 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 332
I20260620 20:21:59.085237 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:21:59.085990 32686 external_mini_cluster.cc:1428] Setting key df596a9af6d807f0bea77ae2ff307a32
Jun 20 20:21:59 dist-test-slave-zzwf krb5kdc[32693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986918, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:21:59.103420 462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:59.080437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:35525 (local address 127.31.235.190:36105)
0620 20:21:59.080933 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:59.080938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:59.082091 (+ 1153us) server_negotiation.cc:407] Connection header received
0620 20:21:59.084192 (+ 2101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:59.084199 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:59.084309 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:59.084440 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:59.086489 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.087619 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.090199 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.090618 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.090773 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:59.095468 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:59.095479 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:59.095488 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:59.095541 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:59.099588 (+ 4047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:59.100577 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:59.100587 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:59.100595 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:59.100776 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:59.101245 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:59.101251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:59.101254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:59.101729 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:59.101916 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:59.102645 (+ 729us) server_negotiation.cc:299] Negotiation successful
0620 20:21:59.102934 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":272,"threads_started":1}
I20260620 20:21:59.104063 463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:59.081068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:36105 (local address 127.31.235.129:35525)
0620 20:21:59.081905 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:59.081950 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:59.083883 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:59.084732 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:59.084742 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:59.085580 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:59.086315 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:59.086335 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.087837 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.087852 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:59.090008 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:59.090028 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.090964 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.090974 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:59.091327 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:59.092157 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:59.092194 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:59.095177 (+ 2983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:59.099799 (+ 4622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:59.099806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:59.099821 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:59.100408 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:59.100918 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:59.100921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:59.100923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:59.101105 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:59.101868 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:59.101874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:59.102214 (+ 340us) client_negotiation.cc:769] Sending connection context
0620 20:21:59.102451 (+ 237us) client_negotiation.cc:240] Negotiation successful
0620 20:21:59.102879 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":364,"threads_started":1}
WARNING: no policy specified for kudu/127.31.235.130@KRBTEST.COM; defaulting to no policy
I20260620 20:21:59.105737 460 heartbeater.cc:347] Connected to a master server at 127.31.235.190:36105
I20260620 20:21:59.106113 460 heartbeater.cc:464] Registering TS with master...
I20260620 20:21:59.106952 460 heartbeater.cc:511] Master 127.31.235.190:36105 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.130@KRBTEST.COM" created.
I20260620 20:21:59.109239 32738 ts_manager.cc:194] Registered new tserver with Master: 2c869cfb52484a1ba3b78e1e882c0c01 (127.31.235.129:38771)
I20260620 20:21:59.110766 32738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:35525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
I20260620 20:21:59.132436 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36105
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:35069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:59.465167 466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:59.465477 466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:59.465690 466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:59.473119 466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:59.473238 466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:59.473476 466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:21:59.483316 466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:35069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36105
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:59.484689 466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:59.486166 466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:21:59.499344 472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:59.499315 474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:21:59.499620 471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:21:59.500170 466 server_base.cc:1034] running on GCE node
I20260620 20:21:59.500738 466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:21:59.501690 466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:21:59.502913 466 hybrid_clock.cc:648] HybridClock initialized: now 1781986919502871 us; error 53 us; skew 500 ppm
I20260620 20:21:59.503226 466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:21:59 dist-test-slave-zzwf krb5kdc[32693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986919, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:21:59.506426 466 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:21:59.507886 466 webserver.cc:460] Webserver started at http://127.31.235.130:41549/ using document root <none> and password file <none>
I20260620 20:21:59.508543 466 fs_manager.cc:359] Metadata directory not provided
I20260620 20:21:59.508690 466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:21:59.508991 466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:21:59.511880 466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "376833ead9bc4169b345bbea0ad56060"
format_stamp: "Formatted at 2026-06-20 20:21:59 on dist-test-slave-zzwf"
server_key: "b2f7adf297db696c7b0f80cf88fcbea1"
server_key_iv: "7dc6e9e13e0046f8a1381bab5e0a66f6"
server_key_version: "encryptionkey@0"
I20260620 20:21:59.512619 466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "376833ead9bc4169b345bbea0ad56060"
format_stamp: "Formatted at 2026-06-20 20:21:59 on dist-test-slave-zzwf"
server_key: "b2f7adf297db696c7b0f80cf88fcbea1"
server_key_iv: "7dc6e9e13e0046f8a1381bab5e0a66f6"
server_key_version: "encryptionkey@0"
I20260620 20:21:59.517472 466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260620 20:21:59.520640 481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:59.521920 466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:21:59.522037 466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "376833ead9bc4169b345bbea0ad56060"
format_stamp: "Formatted at 2026-06-20 20:21:59 on dist-test-slave-zzwf"
server_key: "b2f7adf297db696c7b0f80cf88fcbea1"
server_key_iv: "7dc6e9e13e0046f8a1381bab5e0a66f6"
server_key_version: "encryptionkey@0"
I20260620 20:21:59.522192 466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:21:59.559250 466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:21:59.563033 466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:21:59.563264 466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:21:59.564220 466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:21:59.565423 466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:21:59.565517 466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:59.565599 466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:21:59.565644 466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:21:59.597834 466 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:40289
I20260620 20:21:59.597918 593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:40289 every 8 connection(s)
I20260620 20:21:59.599188 466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:21:59.609371 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 466
I20260620 20:21:59.609581 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:21:59.610164 32686 external_mini_cluster.cc:1428] Setting key 98dd87d8bdf143465125aae5a2d6948b
Jun 20 20:21:59 dist-test-slave-zzwf krb5kdc[32693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986919, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:21:59.621095 596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:59.602306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:36105 (local address 127.31.235.130:37197)
0620 20:21:59.602953 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0620 20:21:59.602994 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:21:59.604552 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:21:59.605199 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:21:59.605207 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:21:59.605774 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:21:59.606339 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:59.606356 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.607310 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.607314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:59.608494 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:21:59.608505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.609214 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.609228 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:21:59.609412 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:59.610183 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:21:59.610204 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:21:59.612397 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:21:59.616333 (+ 3936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:59.616340 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:59.616353 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:59.616814 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:59.617248 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:21:59.617251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:21:59.617254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:21:59.617425 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:21:59.618091 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:21:59.618103 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:21:59.618604 (+ 501us) client_negotiation.cc:769] Sending connection context
0620 20:21:59.618953 (+ 349us) client_negotiation.cc:240] Negotiation successful
0620 20:21:59.619535 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":258,"threads_started":1}
I20260620 20:21:59.622320 462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:21:59.601667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:37197 (local address 127.31.235.190:36105)
0620 20:21:59.601867 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 20:21:59.601873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:21:59.603109 (+ 1236us) server_negotiation.cc:407] Connection header received
0620 20:21:59.604785 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:21:59.604789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:21:59.604852 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:21:59.604938 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:21:59.606478 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.607171 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.608611 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:21:59.609021 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:21:59.609150 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:21:59.612597 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:21:59.612606 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:21:59.612613 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:21:59.612656 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:21:59.616187 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:59.616965 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:59.616971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:59.616977 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:59.617128 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:21:59.617564 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:21:59.617571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:21:59.617577 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:21:59.617938 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:21:59.618096 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 20:21:59.621651 (+ 3555us) server_negotiation.cc:299] Negotiation successful
0620 20:21:59.621908 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:21:59.623514 594 heartbeater.cc:347] Connected to a master server at 127.31.235.190:36105
I20260620 20:21:59.623898 594 heartbeater.cc:464] Registering TS with master...
I20260620 20:21:59.624835 594 heartbeater.cc:511] Master 127.31.235.190:36105 requested a full tablet report, sending...
I20260620 20:21:59.626732 32738 ts_manager.cc:194] Registered new tserver with Master: 376833ead9bc4169b345bbea0ad56060 (127.31.235.130:40289)
I20260620 20:21:59.628006 32738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:37197
WARNING: no policy specified for kudu/127.31.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
I20260620 20:21:59.653071 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36105
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:35069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:21:59.975862 599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:21:59.976212 599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:21:59.976383 599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:21:59.983608 599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:21:59.983765 599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:21:59.983960 599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:21:59.993999 599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:35069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36105
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:21:59.995304 599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:21:59.996699 599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:00.011365 605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:00.011391 607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:00.011350 604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:00.012566 599 server_base.cc:1034] running on GCE node
I20260620 20:22:00.013222 599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:00.014302 599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:00.015511 599 hybrid_clock.cc:648] HybridClock initialized: now 1781986920015456 us; error 42 us; skew 500 ppm
I20260620 20:22:00.015877 599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:00 dist-test-slave-zzwf krb5kdc[32693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986920, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:00.019163 599 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:22:00.020547 599 webserver.cc:460] Webserver started at http://127.31.235.131:45685/ using document root <none> and password file <none>
I20260620 20:22:00.021229 599 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:00.021373 599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:00.021739 599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:00.024801 599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "e4e9d3cb4a6e462b9f3371fa0386713e"
format_stamp: "Formatted at 2026-06-20 20:22:00 on dist-test-slave-zzwf"
server_key: "9a9eaa99b2da80b512b71e237aba9b4d"
server_key_iv: "2712d65bc4f74a3153b1fd039ee1a103"
server_key_version: "encryptionkey@0"
I20260620 20:22:00.025615 599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "e4e9d3cb4a6e462b9f3371fa0386713e"
format_stamp: "Formatted at 2026-06-20 20:22:00 on dist-test-slave-zzwf"
server_key: "9a9eaa99b2da80b512b71e237aba9b4d"
server_key_iv: "2712d65bc4f74a3153b1fd039ee1a103"
server_key_version: "encryptionkey@0"
I20260620 20:22:00.030653 599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 20:22:00.034082 614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:00.035429 599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:22:00.035624 599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "e4e9d3cb4a6e462b9f3371fa0386713e"
format_stamp: "Formatted at 2026-06-20 20:22:00 on dist-test-slave-zzwf"
server_key: "9a9eaa99b2da80b512b71e237aba9b4d"
server_key_iv: "2712d65bc4f74a3153b1fd039ee1a103"
server_key_version: "encryptionkey@0"
I20260620 20:22:00.035837 599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:00.051918 599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:00.056025 599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:00.056324 599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:00.057284 599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:00.058596 599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:00.058698 599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:00.058861 599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:00.058934 599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:00.092404 599 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:45503
I20260620 20:22:00.092422 726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:45503 every 8 connection(s)
I20260620 20:22:00.093755 599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:22:00.099900 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 599
I20260620 20:22:00.100106 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:22:00.100543 32686 external_mini_cluster.cc:1428] Setting key b0b480b398f0aa9f389d34095090b167
Jun 20 20:22:00 dist-test-slave-zzwf krb5kdc[32693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986920, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:00.113000 462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.096393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:37495 (local address 127.31.235.190:36105)
0620 20:22:00.096696 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:00.096701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:00.097883 (+ 1182us) server_negotiation.cc:407] Connection header received
0620 20:22:00.099769 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:00.099774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:00.099859 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:00.099968 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:00.101665 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.102443 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.103960 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.104350 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.104484 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.107711 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:00.107719 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:00.107729 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:00.107759 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:00.110181 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:00.110872 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:00.110882 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:00.110885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:00.111004 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:00.111414 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:00.111418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:00.111422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:00.111665 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:00.111815 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:00.112454 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 20:22:00.112657 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:22:00.114005 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.096984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:36105 (local address 127.31.235.131:37495)
0620 20:22:00.097730 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:00.097773 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:00.099444 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:00.100233 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:00.100246 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:00.100865 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:00.101416 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:00.101432 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.102601 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.102607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:00.103817 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:00.103828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.104567 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.104576 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:00.104822 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.105431 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:00.105452 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:00.107518 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:00.110315 (+ 2797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:00.110325 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:00.110341 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:00.110746 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:00.111113 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:00.111119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:00.111123 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:00.111298 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:00.111790 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:00.111797 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:00.112116 (+ 319us) client_negotiation.cc:769] Sending connection context
0620 20:22:00.112371 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 20:22:00.112868 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":275,"threads_started":1}
I20260620 20:22:00.114149 460 heartbeater.cc:503] Master 127.31.235.190:36105 was elected leader, sending a full tablet report...
I20260620 20:22:00.115806 727 heartbeater.cc:347] Connected to a master server at 127.31.235.190:36105
I20260620 20:22:00.116096 727 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:00.116852 727 heartbeater.cc:511] Master 127.31.235.190:36105 requested a full tablet report, sending...
I20260620 20:22:00.118156 32737 ts_manager.cc:194] Registered new tserver with Master: e4e9d3cb4a6e462b9f3371fa0386713e (127.31.235.131:45503)
I20260620 20:22:00.119004 32737 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:37495
I20260620 20:22:00.128392 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:22:00 dist-test-slave-zzwf krb5kdc[32693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986920, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:22:00 dist-test-slave-zzwf krb5kdc[32693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986920, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:22:00.173658 734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.156371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44412 (local address 127.31.235.129:38771)
0620 20:22:00.157038 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:00.157049 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:00.157082 (+ 33us) server_negotiation.cc:407] Connection header received
0620 20:22:00.157282 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:00.157290 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:00.157528 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:00.157730 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:00.159129 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.160126 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.161412 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.162061 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.162320 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.166797 (+ 4477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:00.166810 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:00.166825 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:00.166870 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:00.170472 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:00.171351 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:00.171359 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:00.171362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:00.171498 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:00.171902 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:00.171909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:00.171912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:00.172287 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:00.172607 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:00.172921 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 20:22:00.173193 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":332,"threads_started":1}
W20260620 20:22:00.175088 375 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44412
Jun 20 20:22:00 dist-test-slave-zzwf krb5kdc[32693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:22:00 dist-test-slave-zzwf krb5kdc[32693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:22:00.222435 734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.208036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44422 (local address 127.31.235.129:38771)
0620 20:22:00.208314 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:00.208323 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:00.208672 (+ 349us) server_negotiation.cc:407] Connection header received
0620 20:22:00.208910 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:00.208914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:00.208982 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:00.209087 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:00.210490 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.211177 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.212385 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.212813 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.212944 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.217090 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:00.217097 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:00.217101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:00.217138 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:00.219853 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:00.220521 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:00.220527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:00.220531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:00.220657 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:00.221049 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:00.221054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:00.221057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:00.221309 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:00.221585 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:00.221900 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 20:22:00.222115 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260620 20:22:00.224030 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 332
I20260620 20:22:00.237022 734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.224914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44426 (local address 127.31.235.129:38771)
0620 20:22:00.225121 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:00.225130 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:00.225523 (+ 393us) server_negotiation.cc:407] Connection header received
0620 20:22:00.225728 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:00.225733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:00.225804 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:00.225900 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:00.227135 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.227831 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.229146 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.229666 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.229799 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.231965 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:00.231971 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:00.231976 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:00.232016 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:00.234478 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:00.235128 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:00.235135 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:00.235139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:00.235281 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:00.235633 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:00.235640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:00.235643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:00.235905 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:00.236148 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:00.236447 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 20:22:00.236661 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:22:00.237802 455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:00.324369 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 332
I20260620 20:22:00.343133 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 466
Jun 20 20:22:00 dist-test-slave-zzwf krb5kdc[32693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.130@KRBTEST.COM
I20260620 20:22:00.359688 741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.344015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38492 (local address 127.31.235.130:40289)
0620 20:22:00.344635 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:00.344645 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:00.344677 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:22:00.344747 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:00.344756 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:00.345034 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:00.345201 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:00.346376 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.348063 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.349229 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.350221 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.350376 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.354300 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:00.354312 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:00.354327 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:00.354374 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:00.357228 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:00.357933 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:00.357947 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:00.357949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:00.358076 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:00.358395 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:00.358398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:00.358400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:00.358747 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:00.358966 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:00.359191 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 20:22:00.359380 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":281,"threads_started":1}
I20260620 20:22:00.360920 589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:00.443442 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 466
I20260620 20:22:00.465593 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 599
Jun 20 20:22:00 dist-test-slave-zzwf krb5kdc[32693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.131@KRBTEST.COM
I20260620 20:22:00.482030 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:00.466756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44966 (local address 127.31.235.131:45503)
0620 20:22:00.467464 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:00.467473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:00.467508 (+ 35us) server_negotiation.cc:407] Connection header received
0620 20:22:00.467608 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:00.467615 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:00.467828 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:00.467960 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:00.469176 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.470445 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.471729 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:00.472315 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:00.472421 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:00.476528 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:00.476539 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:00.476555 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:00.476588 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:00.479375 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:00.480031 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:00.480038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:00.480040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:00.480180 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:00.480486 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:00.480489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:00.480491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:00.480844 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:00.481061 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:00.481434 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 20:22:00.481757 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":330,"threads_started":1}
I20260620 20:22:00.483429 722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:00.565742 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 599
I20260620 20:22:00.582127 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 32706
I20260620 20:22:00.583400 300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:00.687201 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 32706
2026-06-20T20:22:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4047 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:00 dist-test-slave-zzwf krb5kdc[748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:00 dist-test-slave-zzwf krb5kdc[748](info): set up 2 sockets
Jun 20 20:22:00 dist-test-slave-zzwf krb5kdc[748](info): commencing operation
krb5kdc: starting...
W20260620 20:22:01.785892 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:22:01 dist-test-slave-zzwf krb5kdc[748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:22:01.970762 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:36473
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:44669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:36473
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:02.296775 761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:02.297123 761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:02.297293 761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:02.305325 761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:02.305482 761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:02.305595 761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:02.305702 761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:02.305784 761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:02.316071 761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:44669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:36473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:36473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:02.317656 761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:02.318990 761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:02.330626 766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:02.330646 767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:02.330638 769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:02.331492 761 server_base.cc:1034] running on GCE node
I20260620 20:22:02.332335 761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:02.333815 761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:02.335045 761 hybrid_clock.cc:648] HybridClock initialized: now 1781986922334995 us; error 45 us; skew 500 ppm
I20260620 20:22:02.335414 761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:02 dist-test-slave-zzwf krb5kdc[748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986922, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:02.345885 761 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:22:02.347693 761 webserver.cc:460] Webserver started at http://127.31.235.190:39563/ using document root <none> and password file <none>
I20260620 20:22:02.348472 761 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:02.348639 761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:02.348984 761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:02.352222 761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "9f8cc73afe7a4267ae11ebfcfcc813ba"
format_stamp: "Formatted at 2026-06-20 20:22:02 on dist-test-slave-zzwf"
server_key: "016b410590d9d714c3505ceeb2b800fe"
server_key_iv: "701d252c965a0e3a5053396a5a75f46e"
server_key_version: "encryptionkey@0"
I20260620 20:22:02.353046 761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "9f8cc73afe7a4267ae11ebfcfcc813ba"
format_stamp: "Formatted at 2026-06-20 20:22:02 on dist-test-slave-zzwf"
server_key: "016b410590d9d714c3505ceeb2b800fe"
server_key_iv: "701d252c965a0e3a5053396a5a75f46e"
server_key_version: "encryptionkey@0"
I20260620 20:22:02.358747 761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:22:02.362442 776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:02.363816 761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:22:02.364015 761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "9f8cc73afe7a4267ae11ebfcfcc813ba"
format_stamp: "Formatted at 2026-06-20 20:22:02 on dist-test-slave-zzwf"
server_key: "016b410590d9d714c3505ceeb2b800fe"
server_key_iv: "701d252c965a0e3a5053396a5a75f46e"
server_key_version: "encryptionkey@0"
I20260620 20:22:02.364235 761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:02.397614 761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:02.402475 761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:02.402812 761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:02.422389 761 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:36473
I20260620 20:22:02.422406 827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:36473 every 8 connection(s)
I20260620 20:22:02.423913 761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:02.427577 828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:02.429332 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 761
I20260620 20:22:02.429600 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:22:02.430415 32686 external_mini_cluster.cc:1428] Setting key 2b416b2fbaf3fd3ee97a76c498922ad4
I20260620 20:22:02.439997 828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba: Bootstrap starting.
I20260620 20:22:02.443840 828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba: Neither blocks nor log segments found. Creating new log.
Jun 20 20:22:02 dist-test-slave-zzwf krb5kdc[748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:02.445004 828 log.cc:824] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:02.448568 828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba: No bootstrap required, opened a new log
I20260620 20:22:02.454901 831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:02.432478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.31.235.190:36473)
0620 20:22:02.433351 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:02.433363 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:02.433405 (+ 42us) server_negotiation.cc:407] Connection header received
0620 20:22:02.435160 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:02.435182 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:02.435558 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:02.435869 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:02.437635 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:02.438933 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:02.440601 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:02.441242 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:02.441375 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:02.446318 (+ 4943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:02.446325 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:02.446339 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:02.446403 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:02.450205 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:02.450916 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:02.450924 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:02.450934 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:02.451136 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:02.451543 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:02.451548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:02.451552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:02.452256 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:02.452566 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:02.453033 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 20:22:02.453479 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":371,"threads_started":1}
I20260620 20:22:02.458657 828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8cc73afe7a4267ae11ebfcfcc813ba" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36473 } }
I20260620 20:22:02.458993 828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:02.459069 828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f8cc73afe7a4267ae11ebfcfcc813ba, State: Initialized, Role: FOLLOWER
I20260620 20:22:02.459532 828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8cc73afe7a4267ae11ebfcfcc813ba" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36473 } }
I20260620 20:22:02.459725 828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:02.459825 828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:02.459944 828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:02.462064 828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8cc73afe7a4267ae11ebfcfcc813ba" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36473 } }
I20260620 20:22:02.462496 828 leader_election.cc:302] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f8cc73afe7a4267ae11ebfcfcc813ba; no voters:
I20260620 20:22:02.463054 828 leader_election.cc:288] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:02.463213 833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:02.464427 833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba [term 1 LEADER]: Becoming Leader. State: Replica: 9f8cc73afe7a4267ae11ebfcfcc813ba, State: Running, Role: LEADER
I20260620 20:22:02.465018 833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8cc73afe7a4267ae11ebfcfcc813ba" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36473 } }
I20260620 20:22:02.465309 828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:02.468000 835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f8cc73afe7a4267ae11ebfcfcc813ba. Latest consensus state: current_term: 1 leader_uuid: "9f8cc73afe7a4267ae11ebfcfcc813ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8cc73afe7a4267ae11ebfcfcc813ba" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36473 } } }
I20260620 20:22:02.468422 835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:02.468330 834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f8cc73afe7a4267ae11ebfcfcc813ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8cc73afe7a4267ae11ebfcfcc813ba" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36473 } } }
I20260620 20:22:02.468611 834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:02.470548 842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:02.474670 842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:02.483884 842 catalog_manager.cc:1347] Generated new cluster ID: 79661ac7f5a940f1b738e2ca73a9e281
I20260620 20:22:02.484018 842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:02.500393 842 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:02.502349 842 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:02.517732 842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f8cc73afe7a4267ae11ebfcfcc813ba: Generated new TSK 0
I20260620 20:22:02.518712 842 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
I20260620 20:22:02.569350 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36473
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:44669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:02.896878 854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:02.897140 854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:02.897236 854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:02.904758 854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:02.904878 854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:02.904994 854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:22:02.915048 854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:44669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36473
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:02.916301 854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:02.917624 854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:02.930950 859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:02.930994 860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:02.930908 862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:02.935139 854 server_base.cc:1034] running on GCE node
I20260620 20:22:02.935758 854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:02.936740 854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:02.937947 854 hybrid_clock.cc:648] HybridClock initialized: now 1781986922937885 us; error 53 us; skew 500 ppm
I20260620 20:22:02.938263 854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:02 dist-test-slave-zzwf krb5kdc[748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986922, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:02.941409 854 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:22:02.942905 854 webserver.cc:460] Webserver started at http://127.31.235.129:38881/ using document root <none> and password file <none>
I20260620 20:22:02.943599 854 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:02.943732 854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:02.944067 854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:02.947060 854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "3bb5626cd6b947f9a4086d63b59aeb6f"
format_stamp: "Formatted at 2026-06-20 20:22:02 on dist-test-slave-zzwf"
server_key: "6be3871a619b8e2fd9dc0930433b2534"
server_key_iv: "eb6115751d72e39e200dc1ebd042042f"
server_key_version: "encryptionkey@0"
I20260620 20:22:02.947826 854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "3bb5626cd6b947f9a4086d63b59aeb6f"
format_stamp: "Formatted at 2026-06-20 20:22:02 on dist-test-slave-zzwf"
server_key: "6be3871a619b8e2fd9dc0930433b2534"
server_key_iv: "eb6115751d72e39e200dc1ebd042042f"
server_key_version: "encryptionkey@0"
I20260620 20:22:02.952754 854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 20:22:02.956185 869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:02.957578 854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:22:02.957751 854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "3bb5626cd6b947f9a4086d63b59aeb6f"
format_stamp: "Formatted at 2026-06-20 20:22:02 on dist-test-slave-zzwf"
server_key: "6be3871a619b8e2fd9dc0930433b2534"
server_key_iv: "eb6115751d72e39e200dc1ebd042042f"
server_key_version: "encryptionkey@0"
I20260620 20:22:02.957929 854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:02.989985 854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:02.994860 854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:02.995167 854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:02.996209 854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:02.997574 854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:02.997648 854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:02.997774 854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:02.997859 854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:03.031919 854 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:40345
I20260620 20:22:03.031957 981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:40345 every 8 connection(s)
I20260620 20:22:03.033290 854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:22:03.037245 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 854
I20260620 20:22:03.037544 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:22:03.038522 32686 external_mini_cluster.cc:1428] Setting key 41c9ad304bb1a405f3f6231a69110f1e
Jun 20 20:22:03 dist-test-slave-zzwf krb5kdc[748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986922, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.235.130@KRBTEST.COM; defaulting to no policy
I20260620 20:22:03.056855 984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:03.035912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:57611 (local address 127.31.235.190:36473)
0620 20:22:03.036402 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:03.036407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:03.037534 (+ 1127us) server_negotiation.cc:407] Connection header received
0620 20:22:03.040135 (+ 2601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:03.040140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:03.040210 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:03.040391 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:03.042597 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:03.043332 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:03.045922 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:03.046627 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:03.046810 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:03.050868 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:03.050875 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:03.050880 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:03.050941 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:03.053728 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:03.054463 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:03.054472 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:03.054475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:03.054599 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:03.055035 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:03.055047 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:03.055049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:03.055306 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:03.055458 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:03.056233 (+ 775us) server_negotiation.cc:299] Negotiation successful
0620 20:22:03.056451 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":278,"threads_started":1}
I20260620 20:22:03.057996 985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:03.036322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:36473 (local address 127.31.235.129:57611)
0620 20:22:03.037316 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:03.037375 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:03.039844 (+ 2469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:03.040667 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:03.040678 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:03.041419 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:03.042295 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:03.042406 (+ 111us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:03.043514 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:03.043522 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:03.045760 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:03.045783 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:03.046890 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:03.046895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:03.047113 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:03.047844 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:03.047868 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:03.050548 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:03.053880 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:03.053890 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:03.053909 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:03.054331 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:03.054720 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:03.054725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:03.054729 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:03.054918 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:03.055433 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:03.055442 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:03.055810 (+ 368us) client_negotiation.cc:769] Sending connection context
0620 20:22:03.056125 (+ 315us) client_negotiation.cc:240] Negotiation successful
0620 20:22:03.056654 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":396,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.130@KRBTEST.COM" created.
I20260620 20:22:03.059741 982 heartbeater.cc:347] Connected to a master server at 127.31.235.190:36473
I20260620 20:22:03.060129 982 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:03.060962 982 heartbeater.cc:511] Master 127.31.235.190:36473 requested a full tablet report, sending...
I20260620 20:22:03.063329 793 ts_manager.cc:194] Registered new tserver with Master: 3bb5626cd6b947f9a4086d63b59aeb6f (127.31.235.129:40345)
I20260620 20:22:03.064743 793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:57611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
I20260620 20:22:03.085014 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36473
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:44669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:03.406713 988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:03.407048 988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:03.407234 988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:03.414403 988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:03.414566 988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:03.414758 988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:22:03.424499 988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:44669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36473
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:03.425798 988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:03.427053 988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:03.439788 993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:03.439811 996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:03.439921 994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:03.440774 988 server_base.cc:1034] running on GCE node
I20260620 20:22:03.441377 988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:03.442327 988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:03.443565 988 hybrid_clock.cc:648] HybridClock initialized: now 1781986923443528 us; error 47 us; skew 500 ppm
I20260620 20:22:03.443898 988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:03 dist-test-slave-zzwf krb5kdc[748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986923, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:03.447537 988 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:22:03.448980 988 webserver.cc:460] Webserver started at http://127.31.235.130:44595/ using document root <none> and password file <none>
I20260620 20:22:03.449728 988 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:03.449831 988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:03.450084 988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:03.452862 988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "32c0ac34805e4b9a9913aab833536c7d"
format_stamp: "Formatted at 2026-06-20 20:22:03 on dist-test-slave-zzwf"
server_key: "b9ea8ad686d5baa67801d4f9d43ef721"
server_key_iv: "f8f9c40f63387662399e0ff62ae1d051"
server_key_version: "encryptionkey@0"
I20260620 20:22:03.453670 988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "32c0ac34805e4b9a9913aab833536c7d"
format_stamp: "Formatted at 2026-06-20 20:22:03 on dist-test-slave-zzwf"
server_key: "b9ea8ad686d5baa67801d4f9d43ef721"
server_key_iv: "f8f9c40f63387662399e0ff62ae1d051"
server_key_version: "encryptionkey@0"
I20260620 20:22:03.458629 988 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:22:03.462244 1003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:03.463580 988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:22:03.463776 988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "32c0ac34805e4b9a9913aab833536c7d"
format_stamp: "Formatted at 2026-06-20 20:22:03 on dist-test-slave-zzwf"
server_key: "b9ea8ad686d5baa67801d4f9d43ef721"
server_key_iv: "f8f9c40f63387662399e0ff62ae1d051"
server_key_version: "encryptionkey@0"
I20260620 20:22:03.463958 988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:03.489545 988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:03.493970 988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:03.494273 988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:03.495321 988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:03.496732 988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:03.496801 988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:03.496928 988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:03.496976 988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:03.530679 988 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:41725
I20260620 20:22:03.530730 1115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:41725 every 8 connection(s)
I20260620 20:22:03.531970 988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:22:03.532487 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 988
I20260620 20:22:03.532701 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:22:03.533279 32686 external_mini_cluster.cc:1428] Setting key 93c0a0fcacff908c522bfed3fe14dd0b
Jun 20 20:22:03 dist-test-slave-zzwf krb5kdc[748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986923, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.131@KRBTEST.COM" created.
I20260620 20:22:03.555924 984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:03.534992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:57443 (local address 127.31.235.190:36473)
0620 20:22:03.535207 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:03.535215 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:03.536515 (+ 1300us) server_negotiation.cc:407] Connection header received
0620 20:22:03.538408 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:03.538414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:03.538500 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:03.538627 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:03.540282 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:03.540977 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:03.543599 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:03.544004 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:03.544101 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:03.548873 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:03.548880 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:03.548884 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:03.548921 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:03.552079 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:03.552805 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:03.552810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:03.552813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:03.552997 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:03.553793 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:03.553799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:03.553802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:03.554201 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:03.554379 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:03.555121 (+ 742us) server_negotiation.cc:299] Negotiation successful
0620 20:22:03.555420 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:22:03.556474 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:03.535597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:36473 (local address 127.31.235.130:57443)
0620 20:22:03.536364 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:03.536402 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:03.538142 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:03.538866 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:03.538874 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:03.539379 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:03.540090 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:03.540116 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:03.541131 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:03.541140 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:03.543423 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:03.543446 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:03.544240 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:03.544250 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:03.544570 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:03.545291 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:03.545326 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:03.548604 (+ 3278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:03.552228 (+ 3624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:03.552235 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:03.552248 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:03.552652 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:03.553211 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:03.553216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:03.553219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:03.553629 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:03.554358 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:03.554364 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:03.554688 (+ 324us) client_negotiation.cc:769] Sending connection context
0620 20:22:03.554929 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 20:22:03.555329 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":311,"threads_started":1}
I20260620 20:22:03.558032 1116 heartbeater.cc:347] Connected to a master server at 127.31.235.190:36473
I20260620 20:22:03.558442 1116 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:03.559331 1116 heartbeater.cc:511] Master 127.31.235.190:36473 requested a full tablet report, sending...
I20260620 20:22:03.560869 793 ts_manager.cc:194] Registered new tserver with Master: 32c0ac34805e4b9a9913aab833536c7d (127.31.235.130:41725)
I20260620 20:22:03.562363 793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:57443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
I20260620 20:22:03.578910 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36473
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:44669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:03.900844 1121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:03.901149 1121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:03.901248 1121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:03.909008 1121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:03.909121 1121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:03.909235 1121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:22:03.918968 1121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:44669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36473
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:03.920222 1121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:03.921460 1121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:03.934391 1127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:03.934351 1126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:03.934533 1129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:03.935269 1121 server_base.cc:1034] running on GCE node
I20260620 20:22:03.935935 1121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:03.936879 1121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:03.938112 1121 hybrid_clock.cc:648] HybridClock initialized: now 1781986923938054 us; error 54 us; skew 500 ppm
I20260620 20:22:03.938428 1121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:03 dist-test-slave-zzwf krb5kdc[748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986923, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:03.941450 1121 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:22:03.942982 1121 webserver.cc:460] Webserver started at http://127.31.235.131:35309/ using document root <none> and password file <none>
I20260620 20:22:03.943678 1121 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:03.943825 1121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:03.944142 1121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:03.947198 1121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "18e3f7de56f945f2a3f1807c0fe0db2f"
format_stamp: "Formatted at 2026-06-20 20:22:03 on dist-test-slave-zzwf"
server_key: "213c3745ae7aa178a35a2b3b6ad89a29"
server_key_iv: "b4d3e6aa16e9dafaeb297b1a5bf91843"
server_key_version: "encryptionkey@0"
I20260620 20:22:03.947984 1121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "18e3f7de56f945f2a3f1807c0fe0db2f"
format_stamp: "Formatted at 2026-06-20 20:22:03 on dist-test-slave-zzwf"
server_key: "213c3745ae7aa178a35a2b3b6ad89a29"
server_key_iv: "b4d3e6aa16e9dafaeb297b1a5bf91843"
server_key_version: "encryptionkey@0"
I20260620 20:22:03.953259 1121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 20:22:03.957257 1136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:03.958710 1121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:03.958907 1121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "18e3f7de56f945f2a3f1807c0fe0db2f"
format_stamp: "Formatted at 2026-06-20 20:22:03 on dist-test-slave-zzwf"
server_key: "213c3745ae7aa178a35a2b3b6ad89a29"
server_key_iv: "b4d3e6aa16e9dafaeb297b1a5bf91843"
server_key_version: "encryptionkey@0"
I20260620 20:22:03.959131 1121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:03.971982 1121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:03.976342 1121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:03.976649 1121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:03.977759 1121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:03.979151 1121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:03.979255 1121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:03.979403 1121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:03.979475 1121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:04.013103 1121 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:43323
I20260620 20:22:04.013146 1248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:43323 every 8 connection(s)
I20260620 20:22:04.014464 1121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:22:04.016185 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 1121
I20260620 20:22:04.016379 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:22:04.016834 32686 external_mini_cluster.cc:1428] Setting key 0b161d6f84508b528970011140f2b003
Jun 20 20:22:04 dist-test-slave-zzwf krb5kdc[748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986923, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:04.035171 984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:04.017304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:45695 (local address 127.31.235.190:36473)
0620 20:22:04.017532 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:04.017541 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:04.018799 (+ 1258us) server_negotiation.cc:407] Connection header received
0620 20:22:04.020561 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:04.020565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:04.020627 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:04.020711 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:04.022120 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.022849 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.024532 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.024913 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.025004 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:04.028474 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:04.028479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:04.028482 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:04.028516 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:04.031434 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:04.032562 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:04.032568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:04.032573 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:04.032744 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:04.033182 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:04.033196 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:04.033200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:04.033612 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:04.033758 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:04.034605 (+ 847us) server_negotiation.cc:299] Negotiation successful
0620 20:22:04.034821 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:22:04.036135 1251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:04.017831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:36473 (local address 127.31.235.131:45695)
0620 20:22:04.018646 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:04.018698 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:04.020332 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:04.020872 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:04.020879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:04.021346 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:04.021984 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:04.022001 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.023059 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.023064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:04.024386 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:04.024404 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.025118 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.025123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:04.025335 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:04.026077 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:04.026103 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:04.028265 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:04.031599 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:04.031608 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:04.031695 (+ 87us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:04.032400 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:04.032866 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:04.032870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:04.032872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:04.033047 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:04.033843 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:04.033851 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:04.034179 (+ 328us) client_negotiation.cc:769] Sending connection context
0620 20:22:04.034464 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 20:22:04.034974 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":333,"threads_started":1}
I20260620 20:22:04.037667 1249 heartbeater.cc:347] Connected to a master server at 127.31.235.190:36473
I20260620 20:22:04.037998 1249 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:04.038682 1249 heartbeater.cc:511] Master 127.31.235.190:36473 requested a full tablet report, sending...
I20260620 20:22:04.040184 793 ts_manager.cc:194] Registered new tserver with Master: 18e3f7de56f945f2a3f1807c0fe0db2f (127.31.235.131:43323)
I20260620 20:22:04.040961 793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:45695
I20260620 20:22:04.044447 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:22:04.063541 984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:04.050640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33728 (local address 127.31.235.190:36473)
0620 20:22:04.050803 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:04.050807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:04.051279 (+ 472us) server_negotiation.cc:407] Connection header received
0620 20:22:04.051405 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:04.051409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:04.051469 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:04.051549 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:04.052908 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.053705 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.054991 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.055450 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.055557 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:04.058075 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:04.058080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:04.058083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:04.058115 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:04.060685 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:04.061416 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:04.061421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:04.061423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:04.061618 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:04.062040 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:04.062044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:04.062047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:04.062452 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:04.062661 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:04.063019 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 20:22:04.063260 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:22:04.068764 982 heartbeater.cc:503] Master 127.31.235.190:36473 was elected leader, sending a full tablet report...
I20260620 20:22:04.071045 793 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:22:04.078615 793 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:22:04.104857 1262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:04.090813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:41725 (local address 127.0.0.1:43646)
0620 20:22:04.092844 (+ 2031us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:04.092866 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:04.093010 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:04.094196 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:04.094208 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:04.094243 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:04.094668 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:04.094683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.097252 (+ 2569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.097260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:04.100316 (+ 3056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:04.100336 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.103888 (+ 3552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.103895 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:04.104109 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:04.104160 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 20:22:04.104288 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 20:22:04.104478 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1883,"thread_start_us":1769,"threads_started":1}
I20260620 20:22:04.105695 1264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:04.091291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43646 (local address 127.31.235.130:41725)
0620 20:22:04.092439 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:04.092448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:04.093150 (+ 702us) server_negotiation.cc:407] Connection header received
0620 20:22:04.093271 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:04.093282 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:04.093634 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:04.093859 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:04.094902 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.097045 (+ 2143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.100552 (+ 3507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.103249 (+ 2697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.103475 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:04.103685 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:04.105163 (+ 1478us) server_negotiation.cc:299] Negotiation successful
0620 20:22:04.105365 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":994,"thread_start_us":1272,"threads_started":1}
I20260620 20:22:04.110029 1263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:04.090956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44296 (local address 127.31.235.131:43323)
0620 20:22:04.102122 (+ 11166us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:04.102136 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:04.102181 (+ 45us) server_negotiation.cc:407] Connection header received
0620 20:22:04.102301 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:04.102314 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:04.102644 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:04.103616 (+ 972us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:04.103904 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.105418 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.107828 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.108846 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.109058 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:04.109230 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:04.109529 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 20:22:04.109683 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10933,"thread_start_us":10740,"threads_started":1}
I20260620 20:22:04.110019 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:04.090760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:43323 (local address 127.0.0.1:44296)
0620 20:22:04.091642 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:04.091694 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:04.091919 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:04.102920 (+ 11001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:04.102927 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:04.102961 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:04.103381 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:04.103392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.105585 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.105590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:04.107586 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:04.107601 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.109067 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.109072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:04.109321 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:04.109345 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:22:04.109456 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 20:22:04.109603 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":50,"thread_start_us":464,"threads_started":1}
I20260620 20:22:04.109948 1051 tablet_service.cc:1467] Processing CreateTablet for tablet dd4be6d0a5ef4fa78173848ccc35cd6d (DEFAULT_TABLE table=test-table [id=3cf59e56f7774b378c1d5a42f0970d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:04.111531 1051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd4be6d0a5ef4fa78173848ccc35cd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:04.112838 1265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:04.092113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:40345 (local address 127.0.0.1:36688)
0620 20:22:04.103893 (+ 11780us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:04.103913 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:04.104040 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:04.104942 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:04.104946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:04.104970 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:04.105311 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:04.105320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.108196 (+ 2876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.108201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:04.110240 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:04.110254 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.112049 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.112055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:04.112217 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:04.112246 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 20:22:04.112348 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 20:22:04.112465 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11625,"mutex_wait_us":1,"thread_start_us":11469,"threads_started":1}
I20260620 20:22:04.112866 1266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:04.092413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36688 (local address 127.31.235.129:40345)
0620 20:22:04.103270 (+ 10857us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:04.103279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:04.104177 (+ 898us) server_negotiation.cc:407] Connection header received
0620 20:22:04.104292 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:04.104302 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:04.104551 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:04.104726 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:04.105507 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.108061 (+ 2554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.110392 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.111829 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.112028 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:04.112188 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:04.112422 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 20:22:04.112578 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10662,"thread_start_us":10881,"threads_started":1}
I20260620 20:22:04.114094 1184 tablet_service.cc:1467] Processing CreateTablet for tablet dd4be6d0a5ef4fa78173848ccc35cd6d (DEFAULT_TABLE table=test-table [id=3cf59e56f7774b378c1d5a42f0970d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:04.115540 1184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd4be6d0a5ef4fa78173848ccc35cd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:04.116921 917 tablet_service.cc:1467] Processing CreateTablet for tablet dd4be6d0a5ef4fa78173848ccc35cd6d (DEFAULT_TABLE table=test-table [id=3cf59e56f7774b378c1d5a42f0970d16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:04.118451 917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd4be6d0a5ef4fa78173848ccc35cd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:04.127786 1267 tablet_bootstrap.cc:492] T dd4be6d0a5ef4fa78173848ccc35cd6d P 32c0ac34805e4b9a9913aab833536c7d: Bootstrap starting.
I20260620 20:22:04.132565 1267 tablet_bootstrap.cc:654] T dd4be6d0a5ef4fa78173848ccc35cd6d P 32c0ac34805e4b9a9913aab833536c7d: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:04.132568 1269 tablet_bootstrap.cc:492] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f: Bootstrap starting.
I20260620 20:22:04.132586 1268 tablet_bootstrap.cc:492] T dd4be6d0a5ef4fa78173848ccc35cd6d P 18e3f7de56f945f2a3f1807c0fe0db2f: Bootstrap starting.
I20260620 20:22:04.134522 1267 log.cc:824] T dd4be6d0a5ef4fa78173848ccc35cd6d P 32c0ac34805e4b9a9913aab833536c7d: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:04.136144 1269 tablet_bootstrap.cc:654] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:04.136911 1268 tablet_bootstrap.cc:654] T dd4be6d0a5ef4fa78173848ccc35cd6d P 18e3f7de56f945f2a3f1807c0fe0db2f: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:04.137183 1269 log.cc:824] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:04.137905 1267 tablet_bootstrap.cc:492] T dd4be6d0a5ef4fa78173848ccc35cd6d P 32c0ac34805e4b9a9913aab833536c7d: No bootstrap required, opened a new log
I20260620 20:22:04.138132 1268 log.cc:824] T dd4be6d0a5ef4fa78173848ccc35cd6d P 18e3f7de56f945f2a3f1807c0fe0db2f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:04.138182 1267 ts_tablet_manager.cc:1397] T dd4be6d0a5ef4fa78173848ccc35cd6d P 32c0ac34805e4b9a9913aab833536c7d: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20260620 20:22:04.140012 1269 tablet_bootstrap.cc:492] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f: No bootstrap required, opened a new log
I20260620 20:22:04.140270 1269 ts_tablet_manager.cc:1397] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 20:22:04.140978 1268 tablet_bootstrap.cc:492] T dd4be6d0a5ef4fa78173848ccc35cd6d P 18e3f7de56f945f2a3f1807c0fe0db2f: No bootstrap required, opened a new log
I20260620 20:22:04.141228 1268 ts_tablet_manager.cc:1397] T dd4be6d0a5ef4fa78173848ccc35cd6d P 18e3f7de56f945f2a3f1807c0fe0db2f: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260620 20:22:04.148242 1269 raft_consensus.cc:348] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3f7de56f945f2a3f1807c0fe0db2f" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 43323 } } peers { permanent_uuid: "32c0ac34805e4b9a9913aab833536c7d" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 41725 } } peers { permanent_uuid: "3bb5626cd6b947f9a4086d63b59aeb6f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 40345 } }
I20260620 20:22:04.148638 1269 raft_consensus.cc:374] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:04.148761 1269 raft_consensus.cc:729] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb5626cd6b947f9a4086d63b59aeb6f, State: Initialized, Role: FOLLOWER
I20260620 20:22:04.148538 1267 raft_consensus.cc:348] T dd4be6d0a5ef4fa78173848ccc35cd6d P 32c0ac34805e4b9a9913aab833536c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3f7de56f945f2a3f1807c0fe0db2f" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 43323 } } peers { permanent_uuid: "32c0ac34805e4b9a9913aab833536c7d" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 41725 } } peers { permanent_uuid: "3bb5626cd6b947f9a4086d63b59aeb6f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 40345 } }
I20260620 20:22:04.148922 1267 raft_consensus.cc:374] T dd4be6d0a5ef4fa78173848ccc35cd6d P 32c0ac34805e4b9a9913aab833536c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:04.149017 1267 raft_consensus.cc:729] T dd4be6d0a5ef4fa78173848ccc35cd6d P 32c0ac34805e4b9a9913aab833536c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c0ac34805e4b9a9913aab833536c7d, State: Initialized, Role: FOLLOWER
I20260620 20:22:04.149439 1269 consensus_queue.cc:260] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3f7de56f945f2a3f1807c0fe0db2f" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 43323 } } peers { permanent_uuid: "32c0ac34805e4b9a9913aab833536c7d" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 41725 } } peers { permanent_uuid: "3bb5626cd6b947f9a4086d63b59aeb6f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 40345 } }
I20260620 20:22:04.149667 1267 consensus_queue.cc:260] T dd4be6d0a5ef4fa78173848ccc35cd6d P 32c0ac34805e4b9a9913aab833536c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3f7de56f945f2a3f1807c0fe0db2f" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 43323 } } peers { permanent_uuid: "32c0ac34805e4b9a9913aab833536c7d" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 41725 } } peers { permanent_uuid: "3bb5626cd6b947f9a4086d63b59aeb6f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 40345 } }
I20260620 20:22:04.151032 1116 heartbeater.cc:503] Master 127.31.235.190:36473 was elected leader, sending a full tablet report...
I20260620 20:22:04.151672 1267 ts_tablet_manager.cc:1428] T dd4be6d0a5ef4fa78173848ccc35cd6d P 32c0ac34805e4b9a9913aab833536c7d: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20260620 20:22:04.151799 1268 raft_consensus.cc:348] T dd4be6d0a5ef4fa78173848ccc35cd6d P 18e3f7de56f945f2a3f1807c0fe0db2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3f7de56f945f2a3f1807c0fe0db2f" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 43323 } } peers { permanent_uuid: "32c0ac34805e4b9a9913aab833536c7d" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 41725 } } peers { permanent_uuid: "3bb5626cd6b947f9a4086d63b59aeb6f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 40345 } }
I20260620 20:22:04.152294 1268 raft_consensus.cc:374] T dd4be6d0a5ef4fa78173848ccc35cd6d P 18e3f7de56f945f2a3f1807c0fe0db2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:04.152434 1268 raft_consensus.cc:729] T dd4be6d0a5ef4fa78173848ccc35cd6d P 18e3f7de56f945f2a3f1807c0fe0db2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18e3f7de56f945f2a3f1807c0fe0db2f, State: Initialized, Role: FOLLOWER
I20260620 20:22:04.153005 1268 consensus_queue.cc:260] T dd4be6d0a5ef4fa78173848ccc35cd6d P 18e3f7de56f945f2a3f1807c0fe0db2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3f7de56f945f2a3f1807c0fe0db2f" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 43323 } } peers { permanent_uuid: "32c0ac34805e4b9a9913aab833536c7d" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 41725 } } peers { permanent_uuid: "3bb5626cd6b947f9a4086d63b59aeb6f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 40345 } }
I20260620 20:22:04.154137 1269 ts_tablet_manager.cc:1428] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260620 20:22:04.154155 1249 heartbeater.cc:503] Master 127.31.235.190:36473 was elected leader, sending a full tablet report...
I20260620 20:22:04.156993 1268 ts_tablet_manager.cc:1428] T dd4be6d0a5ef4fa78173848ccc35cd6d P 18e3f7de56f945f2a3f1807c0fe0db2f: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260620 20:22:04.189896 1273 raft_consensus.cc:482] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:22:04.190240 1273 raft_consensus.cc:504] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3f7de56f945f2a3f1807c0fe0db2f" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 43323 } } peers { permanent_uuid: "32c0ac34805e4b9a9913aab833536c7d" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 41725 } } peers { permanent_uuid: "3bb5626cd6b947f9a4086d63b59aeb6f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 40345 } }
I20260620 20:22:04.192001 1273 leader_election.cc:288] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18e3f7de56f945f2a3f1807c0fe0db2f (127.31.235.131:43323), 32c0ac34805e4b9a9913aab833536c7d (127.31.235.130:41725)
I20260620 20:22:04.201116 1264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:04.192625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:34745 (local address 127.31.235.130:41725)
0620 20:22:04.192855 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:04.192864 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:04.193284 (+ 420us) server_negotiation.cc:407] Connection header received
0620 20:22:04.193424 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:04.193431 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:04.193535 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:04.193619 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:04.194345 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.195372 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.198568 (+ 3196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.199790 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.199946 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:04.200020 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:04.200730 (+ 710us) server_negotiation.cc:299] Negotiation successful
0620 20:22:04.200859 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:22:04.201507 1276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:04.192211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:43323 (local address 127.31.235.129:40341)
0620 20:22:04.192914 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:04.192936 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:04.193108 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:04.193573 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:04.193577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:04.193623 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:04.194070 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:04.194087 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.195543 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.195550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:04.198395 (+ 2845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:04.198412 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.200136 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.200141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:04.200456 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:04.200584 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 20:22:04.200695 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 20:22:04.200914 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":378,"thread_start_us":375,"threads_started":1}
I20260620 20:22:04.201567 1277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:04.192493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:41725 (local address 127.31.235.129:34745)
0620 20:22:04.193180 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:04.193198 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:04.193300 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:04.193638 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:04.193642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:04.193662 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:04.194175 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:04.194188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.195545 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.195550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:04.198395 (+ 2845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:04.198412 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.200129 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.200137 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:04.200450 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:04.200582 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 20:22:04.200695 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 20:22:04.200914 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":288,"thread_start_us":410,"threads_started":1}
I20260620 20:22:04.201115 1263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:04.192342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:40341 (local address 127.31.235.131:43323)
0620 20:22:04.192575 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:04.192582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:04.193052 (+ 470us) server_negotiation.cc:407] Connection header received
0620 20:22:04.193283 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:04.193289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:04.193387 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:04.193516 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:04.194225 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.195372 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.198567 (+ 3195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.199788 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.199948 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:04.200020 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:04.200730 (+ 710us) server_negotiation.cc:299] Negotiation successful
0620 20:22:04.200863 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:22:04.202829 1204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd4be6d0a5ef4fa78173848ccc35cd6d" candidate_uuid: "3bb5626cd6b947f9a4086d63b59aeb6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18e3f7de56f945f2a3f1807c0fe0db2f" is_pre_election: true
I20260620 20:22:04.202944 1071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd4be6d0a5ef4fa78173848ccc35cd6d" candidate_uuid: "3bb5626cd6b947f9a4086d63b59aeb6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32c0ac34805e4b9a9913aab833536c7d" is_pre_election: true
I20260620 20:22:04.203393 1204 raft_consensus.cc:2454] T dd4be6d0a5ef4fa78173848ccc35cd6d P 18e3f7de56f945f2a3f1807c0fe0db2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bb5626cd6b947f9a4086d63b59aeb6f in term 0.
I20260620 20:22:04.203467 1071 raft_consensus.cc:2454] T dd4be6d0a5ef4fa78173848ccc35cd6d P 32c0ac34805e4b9a9913aab833536c7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bb5626cd6b947f9a4086d63b59aeb6f in term 0.
I20260620 20:22:04.204078 871 leader_election.cc:302] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18e3f7de56f945f2a3f1807c0fe0db2f, 3bb5626cd6b947f9a4086d63b59aeb6f; no voters:
I20260620 20:22:04.204478 1273 raft_consensus.cc:2781] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:22:04.204625 1273 raft_consensus.cc:482] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:22:04.204741 1273 raft_consensus.cc:3037] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:04.206996 1273 raft_consensus.cc:504] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3f7de56f945f2a3f1807c0fe0db2f" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 43323 } } peers { permanent_uuid: "32c0ac34805e4b9a9913aab833536c7d" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 41725 } } peers { permanent_uuid: "3bb5626cd6b947f9a4086d63b59aeb6f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 40345 } }
I20260620 20:22:04.207610 1273 leader_election.cc:288] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f [CANDIDATE]: Term 1 election: Requested vote from peers 18e3f7de56f945f2a3f1807c0fe0db2f (127.31.235.131:43323), 32c0ac34805e4b9a9913aab833536c7d (127.31.235.130:41725)
I20260620 20:22:04.208068 1204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd4be6d0a5ef4fa78173848ccc35cd6d" candidate_uuid: "3bb5626cd6b947f9a4086d63b59aeb6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18e3f7de56f945f2a3f1807c0fe0db2f"
I20260620 20:22:04.208285 1204 raft_consensus.cc:3037] T dd4be6d0a5ef4fa78173848ccc35cd6d P 18e3f7de56f945f2a3f1807c0fe0db2f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:04.208276 1071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd4be6d0a5ef4fa78173848ccc35cd6d" candidate_uuid: "3bb5626cd6b947f9a4086d63b59aeb6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32c0ac34805e4b9a9913aab833536c7d"
I20260620 20:22:04.208498 1071 raft_consensus.cc:3037] T dd4be6d0a5ef4fa78173848ccc35cd6d P 32c0ac34805e4b9a9913aab833536c7d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:04.210563 1204 raft_consensus.cc:2454] T dd4be6d0a5ef4fa78173848ccc35cd6d P 18e3f7de56f945f2a3f1807c0fe0db2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bb5626cd6b947f9a4086d63b59aeb6f in term 1.
I20260620 20:22:04.210778 1071 raft_consensus.cc:2454] T dd4be6d0a5ef4fa78173848ccc35cd6d P 32c0ac34805e4b9a9913aab833536c7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bb5626cd6b947f9a4086d63b59aeb6f in term 1.
I20260620 20:22:04.211020 871 leader_election.cc:302] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18e3f7de56f945f2a3f1807c0fe0db2f, 3bb5626cd6b947f9a4086d63b59aeb6f; no voters:
I20260620 20:22:04.211313 1273 raft_consensus.cc:2781] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:04.211709 1273 raft_consensus.cc:686] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f [term 1 LEADER]: Becoming Leader. State: Replica: 3bb5626cd6b947f9a4086d63b59aeb6f, State: Running, Role: LEADER
I20260620 20:22:04.212174 1273 consensus_queue.cc:237] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3f7de56f945f2a3f1807c0fe0db2f" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 43323 } } peers { permanent_uuid: "32c0ac34805e4b9a9913aab833536c7d" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 41725 } } peers { permanent_uuid: "3bb5626cd6b947f9a4086d63b59aeb6f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 40345 } }
I20260620 20:22:04.216536 793 catalog_manager.cc:5495] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bb5626cd6b947f9a4086d63b59aeb6f (127.31.235.129). New cstate: current_term: 1 leader_uuid: "3bb5626cd6b947f9a4086d63b59aeb6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e3f7de56f945f2a3f1807c0fe0db2f" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 43323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32c0ac34805e4b9a9913aab833536c7d" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 41725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bb5626cd6b947f9a4086d63b59aeb6f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 40345 } health_report { overall_health: HEALTHY } } }
Jun 20 20:22:04 dist-test-slave-zzwf krb5kdc[748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:22:04.243525 1266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:04.228788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36698 (local address 127.31.235.129:40345)
0620 20:22:04.229001 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:04.229009 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:04.229546 (+ 537us) server_negotiation.cc:407] Connection header received
0620 20:22:04.229735 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:04.229738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:04.229808 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:04.229890 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:04.231312 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.232006 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.233170 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.233646 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.233746 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:04.237824 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:04.237839 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:04.237856 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:04.237888 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:04.240716 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:04.241426 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:04.241434 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:04.241436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:04.241654 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:04.242104 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:04.242108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:04.242110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:04.242460 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:04.242705 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:04.242985 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 20:22:04.243150 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 20:22:04.266062 1250 tablet.cc:2369] T dd4be6d0a5ef4fa78173848ccc35cd6d P 18e3f7de56f945f2a3f1807c0fe0db2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 20:22:04 dist-test-slave-zzwf krb5kdc[748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986924, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260620 20:22:04.284266 1117 tablet.cc:2369] T dd4be6d0a5ef4fa78173848ccc35cd6d P 32c0ac34805e4b9a9913aab833536c7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:22:04.285849 983 tablet.cc:2369] T dd4be6d0a5ef4fa78173848ccc35cd6d P 3bb5626cd6b947f9a4086d63b59aeb6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 20:22:04 dist-test-slave-zzwf krb5kdc[748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986924, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:22:04.297083 1266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:04.283492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36702 (local address 127.31.235.129:40345)
0620 20:22:04.283745 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:04.283751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:04.284067 (+ 316us) server_negotiation.cc:407] Connection header received
0620 20:22:04.284270 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:04.284274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:04.284339 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:04.284449 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:04.285863 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.286679 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.287965 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.288363 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.288498 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:04.292300 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:04.292305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:04.292309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:04.292339 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:04.294621 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:04.295203 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:04.295211 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:04.295219 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:04.295339 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:04.295745 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:04.295750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:04.295754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:04.296005 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:04.296294 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:04.296534 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 20:22:04.296750 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 20:22:04 dist-test-slave-zzwf krb5kdc[748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:22:04 dist-test-slave-zzwf krb5kdc[748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:22:04.340574 1266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:04.327385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36718 (local address 127.31.235.129:40345)
0620 20:22:04.327611 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:04.327616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:04.327836 (+ 220us) server_negotiation.cc:407] Connection header received
0620 20:22:04.327978 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:04.327982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:04.328057 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:04.328141 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:04.329484 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.330284 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.331496 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.331945 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.332036 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:04.335858 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:04.335865 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:04.335868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:04.335909 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:04.338432 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:04.339057 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:04.339060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:04.339062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:04.339185 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:04.339521 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:04.339524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:04.339526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:04.339761 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:04.339936 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:04.340164 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 20:22:04.340327 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260620 20:22:04.342352 897 tablet_service.cc:2908] Rejecting scan request for tablet dd4be6d0a5ef4fa78173848ccc35cd6d: Uninitialized: clean time has not yet been initialized
I20260620 20:22:04.345343 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 854
I20260620 20:22:04.357959 1266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:04.346441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36734 (local address 127.31.235.129:40345)
0620 20:22:04.346643 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:04.346650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:04.346848 (+ 198us) server_negotiation.cc:407] Connection header received
0620 20:22:04.347044 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:04.347049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:04.347117 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:04.347214 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:04.348466 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.349177 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.350423 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.350849 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.350982 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:04.353043 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:04.353050 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:04.353054 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:04.353093 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:04.355403 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:04.356039 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:04.356046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:04.356049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:04.356176 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:04.356550 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:04.356555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:04.356558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:04.356804 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:04.357051 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:04.357350 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 20:22:04.357623 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:22:04.358664 977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:04.451447 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 854
I20260620 20:22:04.469220 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 988
Jun 20 20:22:04 dist-test-slave-zzwf krb5kdc[748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.130@KRBTEST.COM
I20260620 20:22:04.484658 1264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:04.470185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.31.235.130:41725)
0620 20:22:04.470382 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:04.470386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:04.470406 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:22:04.470501 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:04.470506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:04.470573 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:04.470670 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:04.471918 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.472734 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.474417 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.474843 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.474992 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:04.479094 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:04.479109 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:04.479126 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:04.479162 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:04.481891 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:04.482672 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:04.482679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:04.482682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:04.482803 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:04.483214 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:04.483218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:04.483220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:04.483578 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:04.483813 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:04.484110 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 20:22:04.484286 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 20:22:04.485357 1111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:04.576256 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 988
I20260620 20:22:04.593029 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 1121
Jun 20 20:22:04 dist-test-slave-zzwf krb5kdc[748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.131@KRBTEST.COM
I20260620 20:22:04.607738 1263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:04.593919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44312 (local address 127.31.235.131:43323)
0620 20:22:04.594105 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:04.594113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:04.594135 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:22:04.594352 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:04.594358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:04.594441 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:04.594550 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:04.595758 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.596488 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.597869 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:04.598276 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:04.598429 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:04.602536 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:04.602550 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:04.602567 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:04.602605 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:04.605198 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:04.605954 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:04.605962 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:04.605964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:04.606100 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:04.606481 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:04.606485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:04.606487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:04.606837 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:04.607062 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:04.607331 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 20:22:04.607492 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:22:04.608359 1244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:04.697926 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 1121
I20260620 20:22:04.714192 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 761
I20260620 20:22:04.715329 823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:04.823674 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 761
2026-06-20T20:22:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4133 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:04 dist-test-slave-zzwf krb5kdc[1297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:04 dist-test-slave-zzwf krb5kdc[1297](info): set up 2 sockets
Jun 20 20:22:04 dist-test-slave-zzwf krb5kdc[1297](info): commencing operation
krb5kdc: starting...
W20260620 20:22:05.929987 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:22:06 dist-test-slave-zzwf krb5kdc[1297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:22:06.111209 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:40921
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:42245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:40921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:06.438560 1310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:06.438848 1310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:06.439059 1310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:06.446241 1310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:06.446346 1310 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 20:22:06.446431 1310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:06.446523 1310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:06.446588 1310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:06.446674 1310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:06.456955 1310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:42245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:40921
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:40921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:06.458319 1310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:06.459697 1310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:06.471938 1316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:06.471871 1318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:06.471972 1315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:06.472777 1310 server_base.cc:1034] running on GCE node
I20260620 20:22:06.473675 1310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:06.475173 1310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:06.476408 1310 hybrid_clock.cc:648] HybridClock initialized: now 1781986926476343 us; error 36 us; skew 500 ppm
I20260620 20:22:06.476715 1310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:06 dist-test-slave-zzwf krb5kdc[1297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986926, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:06.479902 1310 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:22:06.481389 1310 webserver.cc:460] Webserver started at http://127.31.235.190:33045/ using document root <none> and password file <none>
I20260620 20:22:06.482147 1310 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:06.482265 1310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:06.482616 1310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:06.485668 1310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "3fdc64dd9c544aafbccd0277b18972ce"
format_stamp: "Formatted at 2026-06-20 20:22:06 on dist-test-slave-zzwf"
server_key: "7da6e86cfb30ec0d93a8885d6f7f76f1"
server_key_iv: "9012124e3b48a2ef2bd00b67aea80088"
server_key_version: "encryptionkey@0"
I20260620 20:22:06.486475 1310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "3fdc64dd9c544aafbccd0277b18972ce"
format_stamp: "Formatted at 2026-06-20 20:22:06 on dist-test-slave-zzwf"
server_key: "7da6e86cfb30ec0d93a8885d6f7f76f1"
server_key_iv: "9012124e3b48a2ef2bd00b67aea80088"
server_key_version: "encryptionkey@0"
I20260620 20:22:06.491940 1310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:22:06.495572 1325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:06.497030 1310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:22:06.497253 1310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "3fdc64dd9c544aafbccd0277b18972ce"
format_stamp: "Formatted at 2026-06-20 20:22:06 on dist-test-slave-zzwf"
server_key: "7da6e86cfb30ec0d93a8885d6f7f76f1"
server_key_iv: "9012124e3b48a2ef2bd00b67aea80088"
server_key_version: "encryptionkey@0"
I20260620 20:22:06.497493 1310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:06.532503 1310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:06.536641 1310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:06.536944 1310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:06.558722 1310 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:40921
I20260620 20:22:06.558743 1386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:40921 every 8 connection(s)
I20260620 20:22:06.560184 1310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:06.564131 1387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:06.569545 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 1310
I20260620 20:22:06.569775 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:22:06.570405 32686 external_mini_cluster.cc:1428] Setting key 578cc246d11ac627b982a27745555cdb
I20260620 20:22:06.574838 1387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce: Bootstrap starting.
I20260620 20:22:06.578495 1387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:06.579689 1387 log.cc:824] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:06.582899 1387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce: No bootstrap required, opened a new log
Jun 20 20:22:06 dist-test-slave-zzwf krb5kdc[1297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:06.591183 1387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fdc64dd9c544aafbccd0277b18972ce" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 40921 } }
I20260620 20:22:06.591596 1387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:06.591741 1387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fdc64dd9c544aafbccd0277b18972ce, State: Initialized, Role: FOLLOWER
I20260620 20:22:06.592470 1387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fdc64dd9c544aafbccd0277b18972ce" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 40921 } }
I20260620 20:22:06.592756 1387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:06.592892 1387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:06.593070 1387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:06.594136 1390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:06.572398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48240 (local address 127.31.235.190:40921)
0620 20:22:06.573270 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:06.573282 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:06.573324 (+ 42us) server_negotiation.cc:407] Connection header received
0620 20:22:06.575126 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:06.575148 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:06.575527 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:06.575846 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:06.577629 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:06.578962 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:06.580666 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:06.581435 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:06.581662 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:06.586316 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:06.586324 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:06.586345 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:06.586387 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:06.589607 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:06.590344 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:06.590351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:06.590358 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:06.590521 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:06.590905 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:06.590912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:06.590915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:06.591459 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:06.591755 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:06.592207 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 20:22:06.592637 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":430,"threads_started":1}
I20260620 20:22:06.595777 1387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fdc64dd9c544aafbccd0277b18972ce" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 40921 } }
I20260620 20:22:06.596295 1387 leader_election.cc:302] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fdc64dd9c544aafbccd0277b18972ce; no voters:
I20260620 20:22:06.596943 1387 leader_election.cc:288] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:06.597088 1392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:06.597862 1392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce [term 1 LEADER]: Becoming Leader. State: Replica: 3fdc64dd9c544aafbccd0277b18972ce, State: Running, Role: LEADER
I20260620 20:22:06.598387 1392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fdc64dd9c544aafbccd0277b18972ce" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 40921 } }
I20260620 20:22:06.599274 1387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:06.601253 1393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fdc64dd9c544aafbccd0277b18972ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fdc64dd9c544aafbccd0277b18972ce" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 40921 } } }
I20260620 20:22:06.601697 1393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:06.601200 1394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fdc64dd9c544aafbccd0277b18972ce. Latest consensus state: current_term: 1 leader_uuid: "3fdc64dd9c544aafbccd0277b18972ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fdc64dd9c544aafbccd0277b18972ce" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 40921 } } }
I20260620 20:22:06.601866 1394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:06.606428 1399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:06.611541 1399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:06.620508 1399 catalog_manager.cc:1347] Generated new cluster ID: 9f061d20a6284fec8c8a81b0c43ad96c
I20260620 20:22:06.620638 1399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:06.629330 1399 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:06.630704 1399 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:06.638350 1399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce: Generated new TSK 0
I20260620 20:22:06.639237 1399 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
I20260620 20:22:06.693857 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:40921
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:42245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:07.042052 1413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:07.042320 1413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:07.042500 1413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:07.049559 1413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:07.049664 1413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:07.049839 1413 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:22:07.049969 1413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:22:07.059754 1413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:42245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.235.190:40921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:07.061053 1413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:07.062395 1413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:07.075548 1418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:07.075490 1419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:07.075527 1421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:07.076345 1413 server_base.cc:1034] running on GCE node
I20260620 20:22:07.077003 1413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:07.078065 1413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:07.079280 1413 hybrid_clock.cc:648] HybridClock initialized: now 1781986927079199 us; error 54 us; skew 500 ppm
I20260620 20:22:07.079630 1413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:07 dist-test-slave-zzwf krb5kdc[1297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986927, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:07.082995 1413 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:22:07.084473 1413 webserver.cc:460] Webserver started at http://127.31.235.129:37579/ using document root <none> and password file <none>
I20260620 20:22:07.085122 1413 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:07.085209 1413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:07.085476 1413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:07.088634 1413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "e44674c3cb9f4d009494d3bb4c54d0fb"
format_stamp: "Formatted at 2026-06-20 20:22:07 on dist-test-slave-zzwf"
server_key: "6cf71a0514eacf51963d1b7c0c66015c"
server_key_iv: "e871d5acee7de2845594b81bda9bae3a"
server_key_version: "encryptionkey@0"
I20260620 20:22:07.089445 1413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "e44674c3cb9f4d009494d3bb4c54d0fb"
format_stamp: "Formatted at 2026-06-20 20:22:07 on dist-test-slave-zzwf"
server_key: "6cf71a0514eacf51963d1b7c0c66015c"
server_key_iv: "e871d5acee7de2845594b81bda9bae3a"
server_key_version: "encryptionkey@0"
I20260620 20:22:07.094772 1413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 20:22:07.098070 1428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:07.099462 1413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:22:07.099632 1413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "e44674c3cb9f4d009494d3bb4c54d0fb"
format_stamp: "Formatted at 2026-06-20 20:22:07 on dist-test-slave-zzwf"
server_key: "6cf71a0514eacf51963d1b7c0c66015c"
server_key_iv: "e871d5acee7de2845594b81bda9bae3a"
server_key_version: "encryptionkey@0"
I20260620 20:22:07.099848 1413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:07.126442 1413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:07.130944 1413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:07.131246 1413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:07.133947 1413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:07.134059 1413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:07.134224 1413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:07.134351 1413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Jun 20 20:22:07 dist-test-slave-zzwf krb5kdc[1297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986927, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:07.158754 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:07.134513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:37355 (local address 127.31.235.190:40921)
0620 20:22:07.135311 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:07.135319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:07.136333 (+ 1014us) server_negotiation.cc:407] Connection header received
0620 20:22:07.139424 (+ 3091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:07.139431 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:07.139537 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:07.139642 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:07.142944 (+ 3302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:07.144229 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:07.146606 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:07.147190 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:07.147330 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:07.151311 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:07.151317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:07.151320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:07.151353 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:07.154311 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:07.155413 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:07.155419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:07.155423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:07.155595 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:07.156272 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:07.156277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:07.156280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:07.156653 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:07.156837 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:07.158299 (+ 1462us) server_negotiation.cc:299] Negotiation successful
0620 20:22:07.158510 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":520,"threads_started":1}
I20260620 20:22:07.160741 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:07.135274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:40921 (local address 127.31.235.129:37355)
0620 20:22:07.136123 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:07.136199 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:07.139027 (+ 2828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:07.140280 (+ 1253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:07.140294 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:07.141545 (+ 1251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:07.142678 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:07.142716 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:07.144419 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:07.144424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:07.146369 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:07.146391 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:07.147425 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:07.147432 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:07.147638 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:07.148566 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:07.148589 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:07.151061 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:07.154467 (+ 3406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:07.154477 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:07.154498 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:07.155236 (+ 738us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:07.155779 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:07.155785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:07.155790 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:07.156097 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:07.156918 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:07.156932 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:07.157569 (+ 637us) client_negotiation.cc:769] Sending connection context
0620 20:22:07.157988 (+ 419us) client_negotiation.cc:240] Negotiation successful
0620 20:22:07.158721 (+ 733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":317,"threads_started":1}
I20260620 20:22:07.181979 1503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:07.166762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:40921 (local address 127.31.235.129:46517)
0620 20:22:07.167342 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:07.167361 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:07.167527 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:07.167868 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:07.167872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:07.168325 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:07.168602 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:07.168610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:07.169614 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:07.169620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:07.171267 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:07.171288 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:07.172395 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:07.172436 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:07.172601 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:07.173311 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:07.173353 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:07.174573 (+ 1220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:07.179146 (+ 4573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:07.179154 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:07.179162 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:07.179627 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:07.180098 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:07.180114 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:07.180117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:07.180247 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:07.181020 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:07.181024 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:07.181191 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 20:22:07.181327 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 20:22:07.181652 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":334,"threads_started":1}
I20260620 20:22:07.182324 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:07.166667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:46517 (local address 127.31.235.190:40921)
0620 20:22:07.166884 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:07.166888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:07.167452 (+ 564us) server_negotiation.cc:407] Connection header received
0620 20:22:07.167654 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:07.167659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:07.167742 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:07.167846 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:07.168743 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:07.169436 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:07.171430 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:07.172199 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:07.172346 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:07.174815 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:07.174825 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:07.174833 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:07.174884 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:07.178976 (+ 4092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:07.179758 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:07.179767 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:07.179773 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:07.179953 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:07.180365 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:07.180373 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:07.180378 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:07.180861 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:07.181045 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:07.181468 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 20:22:07.181822 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 20:22:07.193282 1413 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:37061
I20260620 20:22:07.193311 1548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:37061 every 8 connection(s)
I20260620 20:22:07.194538 1413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:22:07.196758 1549 heartbeater.cc:347] Connected to a master server at 127.31.235.190:40921
I20260620 20:22:07.196992 1549 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:07.197706 1549 heartbeater.cc:511] Master 127.31.235.190:40921 requested a full tablet report, sending...
I20260620 20:22:07.199661 1342 ts_manager.cc:194] Registered new tserver with Master: e44674c3cb9f4d009494d3bb4c54d0fb (127.31.235.129:37061)
I20260620 20:22:07.201140 1342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:37355
I20260620 20:22:07.203344 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 1413
I20260620 20:22:07.203583 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:22:07.204276 32686 external_mini_cluster.cc:1428] Setting key 46dd302f3ec0e57bbc173156264c2b76
WARNING: no policy specified for kudu/127.31.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
I20260620 20:22:07.245450 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:40921
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:42245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:07.570748 1553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:07.571130 1553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:07.571269 1553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:07.582660 1553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:07.582775 1553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:07.582859 1553 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:22:07.582940 1553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:22:07.592706 1553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:42245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.235.190:40921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:07.594045 1553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:07.595373 1553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:07.608642 1558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:07.610338 1553 server_base.cc:1034] running on GCE node
W20260620 20:22:07.610342 1559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:07.610373 1561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:07.611277 1553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:07.612306 1553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:07.613579 1553 hybrid_clock.cc:648] HybridClock initialized: now 1781986927613457 us; error 50 us; skew 500 ppm
I20260620 20:22:07.613874 1553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:07 dist-test-slave-zzwf krb5kdc[1297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986927, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:07.620292 1553 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:22:07.621872 1553 webserver.cc:460] Webserver started at http://127.31.235.130:34257/ using document root <none> and password file <none>
I20260620 20:22:07.622524 1553 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:07.622611 1553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:07.622871 1553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:07.625720 1553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "73b9b1e003d24555b6efc26bc1637f24"
format_stamp: "Formatted at 2026-06-20 20:22:07 on dist-test-slave-zzwf"
server_key: "3459e72fddf5720740e3ebad0b0a4df0"
server_key_iv: "69ce23ed98551f2309d43ee5ecf3be79"
server_key_version: "encryptionkey@0"
I20260620 20:22:07.626432 1553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "73b9b1e003d24555b6efc26bc1637f24"
format_stamp: "Formatted at 2026-06-20 20:22:07 on dist-test-slave-zzwf"
server_key: "3459e72fddf5720740e3ebad0b0a4df0"
server_key_iv: "69ce23ed98551f2309d43ee5ecf3be79"
server_key_version: "encryptionkey@0"
I20260620 20:22:07.631186 1553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 20:22:07.634392 1568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:07.635655 1553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:07.635782 1553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "73b9b1e003d24555b6efc26bc1637f24"
format_stamp: "Formatted at 2026-06-20 20:22:07 on dist-test-slave-zzwf"
server_key: "3459e72fddf5720740e3ebad0b0a4df0"
server_key_iv: "69ce23ed98551f2309d43ee5ecf3be79"
server_key_version: "encryptionkey@0"
I20260620 20:22:07.635944 1553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:07.659368 1553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:07.663767 1553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:07.663993 1553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:07.666723 1553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:07.666803 1553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:07.666963 1553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:07.667049 1553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:22:07 dist-test-slave-zzwf krb5kdc[1297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986927, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:07.694271 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:07.667128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:36511 (local address 127.31.235.190:40921)
0620 20:22:07.667320 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:07.667326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:07.668908 (+ 1582us) server_negotiation.cc:407] Connection header received
0620 20:22:07.671864 (+ 2956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:07.671870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:07.672003 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:07.672132 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:07.674519 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:07.675822 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:07.678276 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:07.679066 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:07.679240 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:07.683985 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:07.683994 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:07.683998 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:07.684046 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:07.687821 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:07.688858 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:07.688864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:07.688868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:07.689035 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:07.689761 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:07.689771 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:07.689777 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:07.690224 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:07.690398 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:07.693700 (+ 3302us) server_negotiation.cc:299] Negotiation successful
0620 20:22:07.693983 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 20:22:07.694496 1577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:07.667755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:40921 (local address 127.31.235.130:36511)
0620 20:22:07.668718 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:07.668771 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:07.671521 (+ 2750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:07.672434 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:07.672451 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:07.673305 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:07.674264 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:07.674299 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:07.676007 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:07.676015 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:07.678100 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:07.678121 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:07.679340 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:07.679348 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:07.679573 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:07.680363 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:07.680388 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:07.683677 (+ 3289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:07.688011 (+ 4334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:07.688025 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:07.688043 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:07.688705 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:07.689181 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:07.689188 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:07.689199 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:07.689577 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:07.690377 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:07.690387 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:07.690918 (+ 531us) client_negotiation.cc:769] Sending connection context
0620 20:22:07.691321 (+ 403us) client_negotiation.cc:240] Negotiation successful
0620 20:22:07.691971 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":411,"threads_started":1}
I20260620 20:22:07.711319 1553 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:36423
I20260620 20:22:07.711392 1687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:36423 every 8 connection(s)
I20260620 20:22:07.712888 1553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:22:07.714151 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 1553
I20260620 20:22:07.714349 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:22:07.714931 32686 external_mini_cluster.cc:1428] Setting key 1e73cd05f7df582d6ac9c187212067da
I20260620 20:22:07.715056 1688 heartbeater.cc:347] Connected to a master server at 127.31.235.190:40921
I20260620 20:22:07.715323 1688 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:07.716205 1688 heartbeater.cc:511] Master 127.31.235.190:40921 requested a full tablet report, sending...
I20260620 20:22:07.718207 1342 ts_manager.cc:194] Registered new tserver with Master: 73b9b1e003d24555b6efc26bc1637f24 (127.31.235.130:36423)
I20260620 20:22:07.719545 1342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:36511
I20260620 20:22:07.722604 1663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:07.702353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:40921 (local address 127.31.235.130:42143)
0620 20:22:07.702938 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:07.702960 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:07.703109 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:07.703673 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:07.703678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:07.704431 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:07.704986 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:07.705007 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:07.706746 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:07.706753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:07.708956 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:07.708982 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:07.710042 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:07.710048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:07.710227 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:07.711124 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:07.711167 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:07.712101 (+ 934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:07.716391 (+ 4290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:07.716399 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:07.716407 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:07.718353 (+ 1946us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:07.718941 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:07.718949 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:07.718955 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:07.719145 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:07.720117 (+ 972us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:07.720125 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:07.720422 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 20:22:07.720613 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 20:22:07.721991 (+ 1378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":4,"thread_start_us":409,"threads_started":1}
I20260620 20:22:07.723315 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:07.702688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:42143 (local address 127.31.235.190:40921)
0620 20:22:07.702906 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:07.702914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:07.703053 (+ 139us) server_negotiation.cc:407] Connection header received
0620 20:22:07.703237 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:07.703245 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:07.703360 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:07.703472 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:07.705210 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:07.706553 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:07.709141 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:07.709810 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:07.709971 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:07.712328 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:07.712335 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:07.712342 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:07.712389 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:07.716173 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:07.718545 (+ 2372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:07.718554 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:07.718560 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:07.718776 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:07.719305 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:07.719313 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:07.719319 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:07.719863 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:07.720031 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:07.722911 (+ 2880us) server_negotiation.cc:299] Negotiation successful
0620 20:22:07.723105 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
WARNING: no policy specified for kudu/127.31.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
I20260620 20:22:07.757346 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:40921
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:42245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:08.082573 1692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:08.082856 1692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:08.083025 1692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:08.090283 1692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:08.090401 1692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:08.090559 1692 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:22:08.090683 1692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:22:08.101377 1692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:42245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.235.190:40921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:08.102808 1692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:08.104314 1692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:08.117985 1698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:08.118059 1697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:08.118038 1700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:08.119019 1692 server_base.cc:1034] running on GCE node
I20260620 20:22:08.119663 1692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:08.120731 1692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:08.121980 1692 hybrid_clock.cc:648] HybridClock initialized: now 1781986928121899 us; error 45 us; skew 500 ppm
I20260620 20:22:08.122359 1692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:08 dist-test-slave-zzwf krb5kdc[1297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986928, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:08.126061 1692 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:22:08.127668 1692 webserver.cc:460] Webserver started at http://127.31.235.131:44657/ using document root <none> and password file <none>
I20260620 20:22:08.128470 1692 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:08.128629 1692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:08.129005 1692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:08.132421 1692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "72bec57a25ec469c81857d193612ac0c"
format_stamp: "Formatted at 2026-06-20 20:22:08 on dist-test-slave-zzwf"
server_key: "f1b7b47829829c3afda150227fedec07"
server_key_iv: "37eda68fbde6aaad4b3c24388b27cb38"
server_key_version: "encryptionkey@0"
I20260620 20:22:08.133375 1692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "72bec57a25ec469c81857d193612ac0c"
format_stamp: "Formatted at 2026-06-20 20:22:08 on dist-test-slave-zzwf"
server_key: "f1b7b47829829c3afda150227fedec07"
server_key_iv: "37eda68fbde6aaad4b3c24388b27cb38"
server_key_version: "encryptionkey@0"
I20260620 20:22:08.139333 1692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:22:08.142936 1707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:08.144250 1692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:22:08.144438 1692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "72bec57a25ec469c81857d193612ac0c"
format_stamp: "Formatted at 2026-06-20 20:22:08 on dist-test-slave-zzwf"
server_key: "f1b7b47829829c3afda150227fedec07"
server_key_iv: "37eda68fbde6aaad4b3c24388b27cb38"
server_key_version: "encryptionkey@0"
I20260620 20:22:08.144671 1692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:08.178181 1692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:08.182534 1692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:08.182819 1692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:08.185369 1692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:08.185515 1692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:08.185686 1692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:08.185758 1692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:22:08 dist-test-slave-zzwf krb5kdc[1297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986928, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:08.204177 1549 heartbeater.cc:503] Master 127.31.235.190:40921 was elected leader, sending a full tablet report...
I20260620 20:22:08.210573 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:08.186273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:40443 (local address 127.31.235.190:40921)
0620 20:22:08.186499 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:08.186509 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:08.187793 (+ 1284us) server_negotiation.cc:407] Connection header received
0620 20:22:08.190010 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:08.190016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:08.190108 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:08.190220 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:08.192436 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.193842 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.197209 (+ 3367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.197887 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.197995 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:08.203243 (+ 5248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:08.203248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:08.203252 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:08.203286 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:08.207205 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:08.207988 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:08.207994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:08.207998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:08.208137 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:08.208570 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:08.208577 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:08.208580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:08.208857 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:08.209001 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:08.209974 (+ 973us) server_negotiation.cc:299] Negotiation successful
0620 20:22:08.210204 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:22:08.211683 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:08.186774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:40921 (local address 127.31.235.131:40443)
0620 20:22:08.187605 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:08.187647 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:08.189722 (+ 2075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:08.190495 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:08.190509 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:08.191306 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:08.192205 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:08.192252 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.194123 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.194132 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:08.196998 (+ 2866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:08.197015 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.198318 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.198324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:08.198627 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:08.199511 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:08.199545 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:08.202978 (+ 3433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:08.207370 (+ 4392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:08.207381 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:08.207401 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:08.207860 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:08.208257 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:08.208262 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:08.208266 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:08.208458 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:08.208993 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:08.209002 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:08.209477 (+ 475us) client_negotiation.cc:769] Sending connection context
0620 20:22:08.209882 (+ 405us) client_negotiation.cc:240] Negotiation successful
0620 20:22:08.210448 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":364,"threads_started":1}
I20260620 20:22:08.235214 1770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:08.218787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:40921 (local address 127.31.235.131:50417)
0620 20:22:08.219596 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:08.219622 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:08.219797 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:08.220232 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:08.220236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:08.220653 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:08.220948 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:08.220957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.222660 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.222668 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:08.224324 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:08.224342 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.225619 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.225629 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:08.225915 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:08.226637 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:08.226668 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:08.228514 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:08.232502 (+ 3988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:08.232510 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:08.232516 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:08.232949 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:08.233411 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:08.233415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:08.233417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:08.233655 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:08.234366 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:08.234371 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:08.234545 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 20:22:08.234659 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 20:22:08.234899 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":415,"threads_started":1}
I20260620 20:22:08.235541 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:08.219295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:50417 (local address 127.31.235.190:40921)
0620 20:22:08.219486 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:08.219496 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:08.219738 (+ 242us) server_negotiation.cc:407] Connection header received
0620 20:22:08.219964 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:08.219973 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:08.220078 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:08.220214 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:08.221089 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.222467 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.224545 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.225151 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.225330 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:08.228711 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:08.228717 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:08.228720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:08.228753 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:08.232333 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:08.233074 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:08.233081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:08.233087 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:08.233248 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:08.233791 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:08.233798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:08.233804 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:08.234214 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:08.234397 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:08.234804 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 20:22:08.235100 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:22:08.244558 1692 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:41165
I20260620 20:22:08.244582 1826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:41165 every 8 connection(s)
I20260620 20:22:08.245879 1692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:22:08.247989 1827 heartbeater.cc:347] Connected to a master server at 127.31.235.190:40921
I20260620 20:22:08.248270 1827 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:08.249234 1827 heartbeater.cc:511] Master 127.31.235.190:40921 requested a full tablet report, sending...
I20260620 20:22:08.251046 1342 ts_manager.cc:194] Registered new tserver with Master: 72bec57a25ec469c81857d193612ac0c (127.31.235.131:41165)
I20260620 20:22:08.252327 1342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:40443
I20260620 20:22:08.257081 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 1692
I20260620 20:22:08.257292 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:22:08.257790 32686 external_mini_cluster.cc:1428] Setting key db9d9e5203a8b610d78b7a0855c7c62d
I20260620 20:22:08.260696 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:22:08 dist-test-slave-zzwf krb5kdc[1297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986928, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:22:08 dist-test-slave-zzwf krb5kdc[1297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986928, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:08.308525 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:08.293006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48248 (local address 127.31.235.190:40921)
0620 20:22:08.293236 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:08.293242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:08.293451 (+ 209us) server_negotiation.cc:407] Connection header received
0620 20:22:08.293679 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:08.293683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:08.293749 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:08.293849 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:08.295450 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.296283 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.297823 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.298249 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.298389 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:08.302891 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:08.302896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:08.302899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:08.302939 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:08.305938 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:08.306778 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:08.306782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:08.306784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:08.306936 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:08.307269 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:08.307273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:08.307275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:08.307546 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:08.307733 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:08.308058 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 20:22:08.308245 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 20:22:08.312795 1342 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:22:08.319581 1342 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:22:08.343739 1844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:08.333730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:41165 (local address 127.0.0.1:46506)
0620 20:22:08.335525 (+ 1795us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:08.335556 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:08.335844 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:08.336611 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:08.336617 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:08.336646 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:08.337074 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:08.337084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.338911 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.338916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:08.341231 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:08.341241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.342875 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.342884 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:08.343052 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:08.343085 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 20:22:08.343153 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 20:22:08.343278 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1622,"mutex_wait_us":97,"spinlock_wait_cycles":1792,"thread_start_us":2310,"threads_started":1}
I20260620 20:22:08.344198 1843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:08.332820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57754 (local address 127.31.235.130:36423)
0620 20:22:08.333750 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:08.333759 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:08.334543 (+ 784us) server_negotiation.cc:407] Connection header received
0620 20:22:08.334868 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:08.334881 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:08.335156 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:08.335316 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:08.336280 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.337775 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.340860 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.342303 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.342511 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:08.342695 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:08.343715 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0620 20:22:08.343911 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":598,"threads_started":1}
I20260620 20:22:08.344617 1840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:08.332159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:37061 (local address 127.0.0.1:60876)
0620 20:22:08.332956 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:08.332998 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:08.333161 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:08.337079 (+ 3918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:08.337083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:08.337114 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:08.337481 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:08.337539 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.339554 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.339560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:08.341889 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:08.341905 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.343719 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.343726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:08.343925 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:08.343955 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 20:22:08.344021 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 20:22:08.344157 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":39,"thread_start_us":788,"threads_started":1}
I20260620 20:22:08.344799 1842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:08.332613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:36423 (local address 127.0.0.1:57754)
0620 20:22:08.334418 (+ 1805us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:08.334444 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:08.334653 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:08.335586 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:08.335593 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:08.335638 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:08.336066 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:08.336079 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.337924 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.337932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:08.340688 (+ 2756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:08.340701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.342513 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.342519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:08.342714 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:08.342777 (+ 63us) client_negotiation.cc:769] Sending connection context
0620 20:22:08.342925 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 20:22:08.343127 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1635,"mutex_wait_us":77,"spinlock_wait_cycles":31616,"thread_start_us":1512,"threads_started":1}
I20260620 20:22:08.345062 1845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:08.334127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46506 (local address 127.31.235.131:41165)
0620 20:22:08.334997 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:08.335011 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:08.335672 (+ 661us) server_negotiation.cc:407] Connection header received
0620 20:22:08.336092 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:08.336106 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:08.336377 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:08.336536 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:08.337357 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.338757 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.341377 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.342634 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.344021 (+ 1387us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:08.344183 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:08.344247 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 20:22:08.344485 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":397,"threads_started":1}
I20260620 20:22:08.345479 1841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:08.332203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60876 (local address 127.31.235.129:37061)
0620 20:22:08.336569 (+ 4366us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:08.336578 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:08.336604 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:22:08.336690 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:08.336697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:08.336890 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:08.337063 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:08.337716 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.339384 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.342058 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.343471 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.344722 (+ 1251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:08.344905 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:08.344974 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 20:22:08.345176 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4203,"thread_start_us":4078,"threads_started":1}
I20260620 20:22:08.348644 1618 tablet_service.cc:1467] Processing CreateTablet for tablet 1d83d4b9187d4ae69bd9cdf8ca774a78 (DEFAULT_TABLE table=test-table [id=95dad7fc87f44d3c9d8dc885291acc61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:08.348639 1479 tablet_service.cc:1467] Processing CreateTablet for tablet 1d83d4b9187d4ae69bd9cdf8ca774a78 (DEFAULT_TABLE table=test-table [id=95dad7fc87f44d3c9d8dc885291acc61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:08.348647 1761 tablet_service.cc:1467] Processing CreateTablet for tablet 1d83d4b9187d4ae69bd9cdf8ca774a78 (DEFAULT_TABLE table=test-table [id=95dad7fc87f44d3c9d8dc885291acc61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:08.350185 1618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d83d4b9187d4ae69bd9cdf8ca774a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:08.350324 1479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d83d4b9187d4ae69bd9cdf8ca774a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:08.350338 1761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d83d4b9187d4ae69bd9cdf8ca774a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:08.363269 1846 tablet_bootstrap.cc:492] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24: Bootstrap starting.
I20260620 20:22:08.365412 1848 tablet_bootstrap.cc:492] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c: Bootstrap starting.
I20260620 20:22:08.366534 1847 tablet_bootstrap.cc:492] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb: Bootstrap starting.
I20260620 20:22:08.367153 1846 tablet_bootstrap.cc:654] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:08.368451 1846 log.cc:824] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:08.369964 1847 tablet_bootstrap.cc:654] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:08.371165 1847 log.cc:824] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:08.371367 1846 tablet_bootstrap.cc:492] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24: No bootstrap required, opened a new log
I20260620 20:22:08.371628 1846 ts_tablet_manager.cc:1397] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 20:22:08.372210 1848 tablet_bootstrap.cc:654] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:08.373694 1848 log.cc:824] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:08.375689 1847 tablet_bootstrap.cc:492] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb: No bootstrap required, opened a new log
I20260620 20:22:08.375958 1847 ts_tablet_manager.cc:1397] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260620 20:22:08.378088 1848 tablet_bootstrap.cc:492] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c: No bootstrap required, opened a new log
I20260620 20:22:08.378362 1848 ts_tablet_manager.cc:1397] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.004s
I20260620 20:22:08.379289 1846 raft_consensus.cc:348] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } }
I20260620 20:22:08.379637 1846 raft_consensus.cc:374] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:08.379731 1846 raft_consensus.cc:729] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73b9b1e003d24555b6efc26bc1637f24, State: Initialized, Role: FOLLOWER
I20260620 20:22:08.380321 1846 consensus_queue.cc:260] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } }
I20260620 20:22:08.381958 1846 ts_tablet_manager.cc:1428] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260620 20:22:08.382393 1688 heartbeater.cc:503] Master 127.31.235.190:40921 was elected leader, sending a full tablet report...
I20260620 20:22:08.386359 1847 raft_consensus.cc:348] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } }
I20260620 20:22:08.386883 1847 raft_consensus.cc:374] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:08.386989 1847 raft_consensus.cc:729] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e44674c3cb9f4d009494d3bb4c54d0fb, State: Initialized, Role: FOLLOWER
I20260620 20:22:08.387593 1847 consensus_queue.cc:260] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } }
I20260620 20:22:08.389271 1847 ts_tablet_manager.cc:1428] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260620 20:22:08.389874 1848 raft_consensus.cc:348] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } }
I20260620 20:22:08.390218 1848 raft_consensus.cc:374] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:08.390355 1848 raft_consensus.cc:729] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72bec57a25ec469c81857d193612ac0c, State: Initialized, Role: FOLLOWER
I20260620 20:22:08.390925 1848 consensus_queue.cc:260] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } }
I20260620 20:22:08.391992 1827 heartbeater.cc:503] Master 127.31.235.190:40921 was elected leader, sending a full tablet report...
I20260620 20:22:08.392637 1848 ts_tablet_manager.cc:1428] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
W20260620 20:22:08.448376 1550 tablet.cc:2369] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:22:08.472052 1689 tablet.cc:2369] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:22:08.497588 1828 tablet.cc:2369] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:22:08.604439 1853 raft_consensus.cc:482] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:22:08.604846 1853 raft_consensus.cc:504] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } }
I20260620 20:22:08.606736 1853 leader_election.cc:288] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72bec57a25ec469c81857d193612ac0c (127.31.235.131:41165), 73b9b1e003d24555b6efc26bc1637f24 (127.31.235.130:36423)
I20260620 20:22:08.613641 1845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:08.607136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:58123 (local address 127.31.235.131:41165)
0620 20:22:08.607340 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:08.607364 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:08.608080 (+ 716us) server_negotiation.cc:407] Connection header received
0620 20:22:08.608244 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:08.608249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:08.608326 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:08.608425 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:08.609034 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.609882 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.611767 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.612565 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.612700 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:08.612804 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:08.613176 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 20:22:08.613325 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:22:08.613806 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:08.606984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:41165 (local address 127.31.235.129:58123)
0620 20:22:08.607926 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:08.607950 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:08.608101 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:08.608450 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:08.608454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:08.608478 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:08.608877 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:08.608886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.610005 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.610011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:08.611572 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:08.611584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.612783 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.612791 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:08.612969 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:08.613017 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 20:22:08.613132 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 20:22:08.613286 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":2,"thread_start_us":547,"threads_started":1}
I20260620 20:22:08.613723 1843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:08.607299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:33339 (local address 127.31.235.130:36423)
0620 20:22:08.607541 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:08.607549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:08.608225 (+ 676us) server_negotiation.cc:407] Connection header received
0620 20:22:08.608455 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:08.608460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:08.608538 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:08.608641 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:08.609258 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.610111 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.611873 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.612652 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.612813 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:08.612886 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:08.613236 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 20:22:08.613387 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 20:22:08.614255 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:08.607260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:36423 (local address 127.31.235.129:33339)
0620 20:22:08.608090 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:08.608109 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:08.608284 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:08.608670 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:08.608674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:08.608694 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:08.609085 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:08.609100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.610267 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.610271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:08.611716 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:08.611731 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.612880 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.612885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:08.613053 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:08.613111 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 20:22:08.613209 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 20:22:08.613332 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":46,"thread_start_us":501,"threads_started":1}
I20260620 20:22:08.615023 1782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d83d4b9187d4ae69bd9cdf8ca774a78" candidate_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72bec57a25ec469c81857d193612ac0c" is_pre_election: true
I20260620 20:22:08.615265 1638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d83d4b9187d4ae69bd9cdf8ca774a78" candidate_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73b9b1e003d24555b6efc26bc1637f24" is_pre_election: true
I20260620 20:22:08.615655 1782 raft_consensus.cc:2454] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e44674c3cb9f4d009494d3bb4c54d0fb in term 0.
I20260620 20:22:08.615751 1638 raft_consensus.cc:2454] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e44674c3cb9f4d009494d3bb4c54d0fb in term 0.
I20260620 20:22:08.616338 1432 leader_election.cc:302] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73b9b1e003d24555b6efc26bc1637f24, e44674c3cb9f4d009494d3bb4c54d0fb; no voters:
I20260620 20:22:08.616734 1853 raft_consensus.cc:2781] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:22:08.616885 1853 raft_consensus.cc:482] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:22:08.616989 1853 raft_consensus.cc:3037] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:08.619158 1853 raft_consensus.cc:504] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } }
I20260620 20:22:08.619750 1853 leader_election.cc:288] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb [CANDIDATE]: Term 1 election: Requested vote from peers 72bec57a25ec469c81857d193612ac0c (127.31.235.131:41165), 73b9b1e003d24555b6efc26bc1637f24 (127.31.235.130:36423)
I20260620 20:22:08.620134 1782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d83d4b9187d4ae69bd9cdf8ca774a78" candidate_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72bec57a25ec469c81857d193612ac0c"
I20260620 20:22:08.620215 1638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d83d4b9187d4ae69bd9cdf8ca774a78" candidate_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73b9b1e003d24555b6efc26bc1637f24"
I20260620 20:22:08.620353 1782 raft_consensus.cc:3037] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:08.620379 1638 raft_consensus.cc:3037] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:08.623484 1638 raft_consensus.cc:2454] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e44674c3cb9f4d009494d3bb4c54d0fb in term 1.
I20260620 20:22:08.623484 1782 raft_consensus.cc:2454] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e44674c3cb9f4d009494d3bb4c54d0fb in term 1.
I20260620 20:22:08.623917 1430 leader_election.cc:302] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72bec57a25ec469c81857d193612ac0c, e44674c3cb9f4d009494d3bb4c54d0fb; no voters:
I20260620 20:22:08.624228 1853 raft_consensus.cc:2781] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:08.624728 1853 raft_consensus.cc:686] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb [term 1 LEADER]: Becoming Leader. State: Replica: e44674c3cb9f4d009494d3bb4c54d0fb, State: Running, Role: LEADER
I20260620 20:22:08.625242 1853 consensus_queue.cc:237] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } }
I20260620 20:22:08.630126 1342 catalog_manager.cc:5495] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb reported cstate change: term changed from 0 to 1, leader changed from <none> to e44674c3cb9f4d009494d3bb4c54d0fb (127.31.235.129). New cstate: current_term: 1 leader_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } health_report { overall_health: UNKNOWN } } }
I20260620 20:22:08.717032 1841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:08.710076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60878 (local address 127.31.235.129:37061)
0620 20:22:08.710346 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:08.710353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:08.710433 (+ 80us) server_negotiation.cc:407] Connection header received
0620 20:22:08.710830 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:08.710835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:08.710935 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:08.711058 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:08.711493 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.712251 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.714342 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.714908 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.715015 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:08.715684 (+ 669us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:08.716266 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:08.716386 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:08.716618 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 20:22:08.716698 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 20:22:08.726212 1782 raft_consensus.cc:1261] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c [term 1 FOLLOWER]: Refusing update from remote peer e44674c3cb9f4d009494d3bb4c54d0fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:08.726325 1638 raft_consensus.cc:1261] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24 [term 1 FOLLOWER]: Refusing update from remote peer e44674c3cb9f4d009494d3bb4c54d0fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:08.727273 1857 consensus_queue.cc:1035] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:08.727717 1853 consensus_queue.cc:1035] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:08.740478 1861 mvcc.cc:205] Tried to move back new op lower bound from 7299018460052824064 to 7299018459652984832. Current Snapshot: MvccSnapshot[applied={T|T < 7299018460052824064}]
I20260620 20:22:08.742811 1862 mvcc.cc:205] Tried to move back new op lower bound from 7299018460052824064 to 7299018459652984832. Current Snapshot: MvccSnapshot[applied={T|T < 7299018460052824064}]
I20260620 20:22:08.757414 1341 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:08.757789 1341 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:08.761382 1341 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce: Sending DeleteTablet for 3 replicas of tablet 1d83d4b9187d4ae69bd9cdf8ca774a78
I20260620 20:22:08.762518 1479 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d83d4b9187d4ae69bd9cdf8ca774a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:08 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:60876
I20260620 20:22:08.762605 1761 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d83d4b9187d4ae69bd9cdf8ca774a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:08 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:46506
I20260620 20:22:08.762890 1618 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d83d4b9187d4ae69bd9cdf8ca774a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:08 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:57754
I20260620 20:22:08.768095 1341 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:22:08.769613 1341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:22:08.780507 1617 tablet_service.cc:1467] Processing CreateTablet for tablet 2e40c0b7d4ac42e682adbcc9207e4fd9 (DEFAULT_TABLE table=test-table [id=3abf491cb5594c98bb9d3d45377ede09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:08.781319 1617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e40c0b7d4ac42e682adbcc9207e4fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:08.782823 1478 tablet_service.cc:1467] Processing CreateTablet for tablet 2e40c0b7d4ac42e682adbcc9207e4fd9 (DEFAULT_TABLE table=test-table [id=3abf491cb5594c98bb9d3d45377ede09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:08.782960 1760 tablet_service.cc:1467] Processing CreateTablet for tablet 2e40c0b7d4ac42e682adbcc9207e4fd9 (DEFAULT_TABLE table=test-table [id=3abf491cb5594c98bb9d3d45377ede09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:08.783391 1760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e40c0b7d4ac42e682adbcc9207e4fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:08.783401 1478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e40c0b7d4ac42e682adbcc9207e4fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:08.789611 1846 tablet_bootstrap.cc:492] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 73b9b1e003d24555b6efc26bc1637f24: Bootstrap starting.
I20260620 20:22:08.790700 1847 tablet_bootstrap.cc:492] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb: Bootstrap starting.
I20260620 20:22:08.791318 1846 tablet_bootstrap.cc:654] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 73b9b1e003d24555b6efc26bc1637f24: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:08.792402 1847 tablet_bootstrap.cc:654] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:08.794458 1848 tablet_bootstrap.cc:492] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 72bec57a25ec469c81857d193612ac0c: Bootstrap starting.
I20260620 20:22:08.796442 1848 tablet_bootstrap.cc:654] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 72bec57a25ec469c81857d193612ac0c: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:08.804301 1848 tablet_bootstrap.cc:492] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 72bec57a25ec469c81857d193612ac0c: No bootstrap required, opened a new log
I20260620 20:22:08.804764 1872 tablet_replica.cc:330] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24: stopping tablet replica
I20260620 20:22:08.805136 1848 ts_tablet_manager.cc:1397] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 72bec57a25ec469c81857d193612ac0c: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20260620 20:22:08.805256 1872 raft_consensus.cc:2229] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:22:08.806394 1848 raft_consensus.cc:348] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 72bec57a25ec469c81857d193612ac0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } }
I20260620 20:22:08.806788 1848 raft_consensus.cc:374] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 72bec57a25ec469c81857d193612ac0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:08.806946 1848 raft_consensus.cc:729] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 72bec57a25ec469c81857d193612ac0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72bec57a25ec469c81857d193612ac0c, State: Initialized, Role: FOLLOWER
I20260620 20:22:08.807350 1848 consensus_queue.cc:260] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 72bec57a25ec469c81857d193612ac0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } }
I20260620 20:22:08.807550 1872 raft_consensus.cc:2258] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:08.807976 1871 tablet_replica.cc:330] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb: stopping tablet replica
I20260620 20:22:08.808418 1871 raft_consensus.cc:2229] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:22:08.809092 1871 raft_consensus.cc:2258] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:08.809919 1848 ts_tablet_manager.cc:1428] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 72bec57a25ec469c81857d193612ac0c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 20:22:08.810170 1872 ts_tablet_manager.cc:1905] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:08.812162 1846 tablet_bootstrap.cc:492] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 73b9b1e003d24555b6efc26bc1637f24: No bootstrap required, opened a new log
I20260620 20:22:08.812168 1847 tablet_bootstrap.cc:492] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb: No bootstrap required, opened a new log
I20260620 20:22:08.812314 1847 ts_tablet_manager.cc:1397] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.005s
I20260620 20:22:08.812305 1846 ts_tablet_manager.cc:1397] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 73b9b1e003d24555b6efc26bc1637f24: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20260620 20:22:08.813076 1846 raft_consensus.cc:348] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 73b9b1e003d24555b6efc26bc1637f24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } }
I20260620 20:22:08.813091 1847 raft_consensus.cc:348] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } }
I20260620 20:22:08.813292 1846 raft_consensus.cc:374] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 73b9b1e003d24555b6efc26bc1637f24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:08.813300 1847 raft_consensus.cc:374] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:08.813421 1847 raft_consensus.cc:729] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e44674c3cb9f4d009494d3bb4c54d0fb, State: Initialized, Role: FOLLOWER
I20260620 20:22:08.813410 1846 raft_consensus.cc:729] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 73b9b1e003d24555b6efc26bc1637f24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73b9b1e003d24555b6efc26bc1637f24, State: Initialized, Role: FOLLOWER
I20260620 20:22:08.813848 1846 consensus_queue.cc:260] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 73b9b1e003d24555b6efc26bc1637f24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } }
I20260620 20:22:08.813884 1847 consensus_queue.cc:260] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } }
I20260620 20:22:08.814133 1853 raft_consensus.cc:482] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:22:08.814255 1853 raft_consensus.cc:504] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } }
I20260620 20:22:08.814399 1846 ts_tablet_manager.cc:1428] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 73b9b1e003d24555b6efc26bc1637f24: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 20:22:08.814935 1853 leader_election.cc:288] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73b9b1e003d24555b6efc26bc1637f24 (127.31.235.130:36423), 72bec57a25ec469c81857d193612ac0c (127.31.235.131:41165)
I20260620 20:22:08.815694 1871 ts_tablet_manager.cc:1905] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:08.815637 1638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e40c0b7d4ac42e682adbcc9207e4fd9" candidate_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73b9b1e003d24555b6efc26bc1637f24" is_pre_election: true
I20260620 20:22:08.815639 1782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e40c0b7d4ac42e682adbcc9207e4fd9" candidate_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72bec57a25ec469c81857d193612ac0c" is_pre_election: true
I20260620 20:22:08.815868 1638 raft_consensus.cc:2454] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 73b9b1e003d24555b6efc26bc1637f24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e44674c3cb9f4d009494d3bb4c54d0fb in term 0.
I20260620 20:22:08.815881 1782 raft_consensus.cc:2454] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 72bec57a25ec469c81857d193612ac0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e44674c3cb9f4d009494d3bb4c54d0fb in term 0.
I20260620 20:22:08.816309 1432 leader_election.cc:302] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73b9b1e003d24555b6efc26bc1637f24, e44674c3cb9f4d009494d3bb4c54d0fb; no voters:
I20260620 20:22:08.816665 1853 raft_consensus.cc:2781] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:22:08.816792 1853 raft_consensus.cc:482] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:22:08.816905 1853 raft_consensus.cc:3037] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:08.817996 1870 tablet_replica.cc:330] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c: stopping tablet replica
I20260620 20:22:08.818424 1870 raft_consensus.cc:2229] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:22:08.819049 1870 raft_consensus.cc:2258] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:08.820955 1853 raft_consensus.cc:504] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } }
I20260620 20:22:08.822077 1638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e40c0b7d4ac42e682adbcc9207e4fd9" candidate_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73b9b1e003d24555b6efc26bc1637f24"
I20260620 20:22:08.822285 1638 raft_consensus.cc:3037] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 73b9b1e003d24555b6efc26bc1637f24 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:08.825124 1638 raft_consensus.cc:2454] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 73b9b1e003d24555b6efc26bc1637f24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e44674c3cb9f4d009494d3bb4c54d0fb in term 1.
I20260620 20:22:08.825276 1872 ts_tablet_manager.cc:1918] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:22:08.825503 1872 log.cc:1196] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/1d83d4b9187d4ae69bd9cdf8ca774a78
I20260620 20:22:08.825608 1432 leader_election.cc:302] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73b9b1e003d24555b6efc26bc1637f24, e44674c3cb9f4d009494d3bb4c54d0fb; no voters:
I20260620 20:22:08.825884 1865 raft_consensus.cc:2781] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:08.826053 1865 raft_consensus.cc:686] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb [term 1 LEADER]: Becoming Leader. State: Replica: e44674c3cb9f4d009494d3bb4c54d0fb, State: Running, Role: LEADER
I20260620 20:22:08.826452 1853 leader_election.cc:288] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb [CANDIDATE]: Term 1 election: Requested vote from peers 73b9b1e003d24555b6efc26bc1637f24 (127.31.235.130:36423), 72bec57a25ec469c81857d193612ac0c (127.31.235.131:41165)
I20260620 20:22:08.826536 1782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e40c0b7d4ac42e682adbcc9207e4fd9" candidate_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72bec57a25ec469c81857d193612ac0c"
I20260620 20:22:08.826362 1865 consensus_queue.cc:237] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } }
I20260620 20:22:08.826728 1782 raft_consensus.cc:3037] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 72bec57a25ec469c81857d193612ac0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:08.826901 1871 ts_tablet_manager.cc:1918] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:22:08.827039 1871 log.cc:1196] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/1d83d4b9187d4ae69bd9cdf8ca774a78
I20260620 20:22:08.826321 1872 ts_tablet_manager.cc:1939] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 73b9b1e003d24555b6efc26bc1637f24: Deleting consensus metadata
I20260620 20:22:08.827543 1871 ts_tablet_manager.cc:1939] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P e44674c3cb9f4d009494d3bb4c54d0fb: Deleting consensus metadata
I20260620 20:22:08.829087 1329 catalog_manager.cc:4841] TS 73b9b1e003d24555b6efc26bc1637f24 (127.31.235.130:36423): tablet 1d83d4b9187d4ae69bd9cdf8ca774a78 (table test-table [id=95dad7fc87f44d3c9d8dc885291acc61]) successfully deleted
I20260620 20:22:08.829401 1870 ts_tablet_manager.cc:1905] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:08.829628 1782 raft_consensus.cc:2454] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 72bec57a25ec469c81857d193612ac0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e44674c3cb9f4d009494d3bb4c54d0fb in term 1.
I20260620 20:22:08.831276 1847 ts_tablet_manager.cc:1428] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20260620 20:22:08.831060 1341 catalog_manager.cc:5495] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb reported cstate change: term changed from 0 to 1, leader changed from <none> to e44674c3cb9f4d009494d3bb4c54d0fb (127.31.235.129). New cstate: current_term: 1 leader_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } health_report { overall_health: UNKNOWN } } }
I20260620 20:22:08.832290 1327 catalog_manager.cc:4841] TS e44674c3cb9f4d009494d3bb4c54d0fb (127.31.235.129:37061): tablet 1d83d4b9187d4ae69bd9cdf8ca774a78 (table test-table [id=95dad7fc87f44d3c9d8dc885291acc61]) successfully deleted
I20260620 20:22:08.836607 1870 ts_tablet_manager.cc:1918] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:22:08.836781 1870 log.cc:1196] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/1d83d4b9187d4ae69bd9cdf8ca774a78
I20260620 20:22:08.837311 1870 ts_tablet_manager.cc:1939] T 1d83d4b9187d4ae69bd9cdf8ca774a78 P 72bec57a25ec469c81857d193612ac0c: Deleting consensus metadata
I20260620 20:22:08.839056 1327 catalog_manager.cc:4841] TS 72bec57a25ec469c81857d193612ac0c (127.31.235.131:41165): tablet 1d83d4b9187d4ae69bd9cdf8ca774a78 (table test-table [id=95dad7fc87f44d3c9d8dc885291acc61]) successfully deleted
Jun 20 20:22:08 dist-test-slave-zzwf krb5kdc[1297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986926, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:08.883270 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:08.868097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:40921 (local address 127.0.0.1:48250)
0620 20:22:08.868781 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:08.868806 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:08.868958 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:08.869410 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:08.869415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:08.870309 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:08.870642 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:08.870660 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.871779 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.871784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:08.873317 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:08.873331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.874144 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.874151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:08.874328 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:08.875040 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:08.875083 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:08.877703 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:08.880646 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:08.880658 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:08.880667 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:08.881074 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:08.881431 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:08.881437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:08.881441 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:08.881656 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:08.882175 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:08.882183 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:08.882349 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 20:22:08.882511 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 20:22:08.882826 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":355,"threads_started":1}
I20260620 20:22:08.883349 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:08.868104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48250 (local address 127.31.235.190:40921)
0620 20:22:08.869110 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:08.869114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:08.869135 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:22:08.869179 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:08.869183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:08.869246 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:08.869334 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:08.870834 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.871648 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.873548 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:08.873965 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:08.874065 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:08.877916 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:08.877921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:08.877924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:08.877950 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:08.880504 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:08.881189 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:08.881197 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:08.881202 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:08.881326 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:08.881782 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:08.881786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:08.881790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:08.882050 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:08.882199 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:08.882614 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 20:22:08.882947 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":921,"spinlock_wait_cycles":11520,"thread_start_us":768,"threads_started":1}
I20260620 20:22:08.888572 1341 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:48250:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 20:22:08.889747 1341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:22:08.896519 1479 tablet_service.cc:1467] Processing CreateTablet for tablet 9b9bafee9c284905b752e53efd9943d1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=490d782a80e74910acfa0e965b1e6559]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:22:08.896512 1618 tablet_service.cc:1467] Processing CreateTablet for tablet 9b9bafee9c284905b752e53efd9943d1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=490d782a80e74910acfa0e965b1e6559]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:22:08.897114 1618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b9bafee9c284905b752e53efd9943d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:08.897121 1479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b9bafee9c284905b752e53efd9943d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:08.904564 1846 tablet_bootstrap.cc:492] T 9b9bafee9c284905b752e53efd9943d1 P 73b9b1e003d24555b6efc26bc1637f24: Bootstrap starting.
I20260620 20:22:08.904575 1847 tablet_bootstrap.cc:492] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb: Bootstrap starting.
I20260620 20:22:08.906013 1846 tablet_bootstrap.cc:654] T 9b9bafee9c284905b752e53efd9943d1 P 73b9b1e003d24555b6efc26bc1637f24: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:08.906421 1847 tablet_bootstrap.cc:654] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:08.907666 1761 tablet_service.cc:1467] Processing CreateTablet for tablet 9b9bafee9c284905b752e53efd9943d1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=490d782a80e74910acfa0e965b1e6559]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:22:08.908248 1761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b9bafee9c284905b752e53efd9943d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:08.910601 1846 tablet_bootstrap.cc:492] T 9b9bafee9c284905b752e53efd9943d1 P 73b9b1e003d24555b6efc26bc1637f24: No bootstrap required, opened a new log
I20260620 20:22:08.910763 1846 ts_tablet_manager.cc:1397] T 9b9bafee9c284905b752e53efd9943d1 P 73b9b1e003d24555b6efc26bc1637f24: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 20:22:08.911516 1846 raft_consensus.cc:348] T 9b9bafee9c284905b752e53efd9943d1 P 73b9b1e003d24555b6efc26bc1637f24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } }
I20260620 20:22:08.911765 1846 raft_consensus.cc:374] T 9b9bafee9c284905b752e53efd9943d1 P 73b9b1e003d24555b6efc26bc1637f24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:08.911854 1846 raft_consensus.cc:729] T 9b9bafee9c284905b752e53efd9943d1 P 73b9b1e003d24555b6efc26bc1637f24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73b9b1e003d24555b6efc26bc1637f24, State: Initialized, Role: FOLLOWER
I20260620 20:22:08.912133 1846 consensus_queue.cc:260] T 9b9bafee9c284905b752e53efd9943d1 P 73b9b1e003d24555b6efc26bc1637f24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } }
I20260620 20:22:08.912566 1852 tablet_replica.cc:439] T 9b9bafee9c284905b752e53efd9943d1 P 73b9b1e003d24555b6efc26bc1637f24: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } } }
I20260620 20:22:08.912907 1852 tablet_replica.cc:442] T 9b9bafee9c284905b752e53efd9943d1 P 73b9b1e003d24555b6efc26bc1637f24: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:22:08.914067 1846 ts_tablet_manager.cc:1428] T 9b9bafee9c284905b752e53efd9943d1 P 73b9b1e003d24555b6efc26bc1637f24: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 20:22:08.915031 1848 tablet_bootstrap.cc:492] T 9b9bafee9c284905b752e53efd9943d1 P 72bec57a25ec469c81857d193612ac0c: Bootstrap starting.
I20260620 20:22:08.916579 1848 tablet_bootstrap.cc:654] T 9b9bafee9c284905b752e53efd9943d1 P 72bec57a25ec469c81857d193612ac0c: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:08.919050 1848 tablet_bootstrap.cc:492] T 9b9bafee9c284905b752e53efd9943d1 P 72bec57a25ec469c81857d193612ac0c: No bootstrap required, opened a new log
I20260620 20:22:08.919231 1848 ts_tablet_manager.cc:1397] T 9b9bafee9c284905b752e53efd9943d1 P 72bec57a25ec469c81857d193612ac0c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 20:22:08.919803 1848 raft_consensus.cc:348] T 9b9bafee9c284905b752e53efd9943d1 P 72bec57a25ec469c81857d193612ac0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } }
I20260620 20:22:08.920019 1848 raft_consensus.cc:374] T 9b9bafee9c284905b752e53efd9943d1 P 72bec57a25ec469c81857d193612ac0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:08.920120 1848 raft_consensus.cc:729] T 9b9bafee9c284905b752e53efd9943d1 P 72bec57a25ec469c81857d193612ac0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72bec57a25ec469c81857d193612ac0c, State: Initialized, Role: FOLLOWER
I20260620 20:22:08.920380 1848 consensus_queue.cc:260] T 9b9bafee9c284905b752e53efd9943d1 P 72bec57a25ec469c81857d193612ac0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } }
I20260620 20:22:08.920809 1854 tablet_replica.cc:439] T 9b9bafee9c284905b752e53efd9943d1 P 72bec57a25ec469c81857d193612ac0c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } } }
I20260620 20:22:08.921147 1854 tablet_replica.cc:442] T 9b9bafee9c284905b752e53efd9943d1 P 72bec57a25ec469c81857d193612ac0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:22:08.921202 1847 tablet_bootstrap.cc:492] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb: No bootstrap required, opened a new log
I20260620 20:22:08.921379 1847 ts_tablet_manager.cc:1397] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.007s
I20260620 20:22:08.922035 1848 ts_tablet_manager.cc:1428] T 9b9bafee9c284905b752e53efd9943d1 P 72bec57a25ec469c81857d193612ac0c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 20:22:08.922125 1847 raft_consensus.cc:348] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } }
I20260620 20:22:08.922394 1847 raft_consensus.cc:374] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:08.922477 1847 raft_consensus.cc:729] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e44674c3cb9f4d009494d3bb4c54d0fb, State: Initialized, Role: FOLLOWER
I20260620 20:22:08.922744 1847 consensus_queue.cc:260] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } }
I20260620 20:22:08.923161 1865 tablet_replica.cc:439] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } } }
I20260620 20:22:08.923448 1865 tablet_replica.cc:442] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:22:08.924283 1847 ts_tablet_manager.cc:1428] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 20:22:09.105454 1865 raft_consensus.cc:482] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:22:09.105783 1865 raft_consensus.cc:504] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } }
I20260620 20:22:09.106482 1865 leader_election.cc:288] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73b9b1e003d24555b6efc26bc1637f24 (127.31.235.130:36423), 72bec57a25ec469c81857d193612ac0c (127.31.235.131:41165)
I20260620 20:22:09.107090 1638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b9bafee9c284905b752e53efd9943d1" candidate_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73b9b1e003d24555b6efc26bc1637f24" is_pre_election: true
I20260620 20:22:09.107082 1782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b9bafee9c284905b752e53efd9943d1" candidate_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72bec57a25ec469c81857d193612ac0c" is_pre_election: true
I20260620 20:22:09.107359 1782 raft_consensus.cc:2454] T 9b9bafee9c284905b752e53efd9943d1 P 72bec57a25ec469c81857d193612ac0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e44674c3cb9f4d009494d3bb4c54d0fb in term 0.
I20260620 20:22:09.107354 1638 raft_consensus.cc:2454] T 9b9bafee9c284905b752e53efd9943d1 P 73b9b1e003d24555b6efc26bc1637f24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e44674c3cb9f4d009494d3bb4c54d0fb in term 0.
I20260620 20:22:09.107805 1430 leader_election.cc:302] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72bec57a25ec469c81857d193612ac0c, e44674c3cb9f4d009494d3bb4c54d0fb; no voters:
I20260620 20:22:09.108136 1865 raft_consensus.cc:2781] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:22:09.108271 1865 raft_consensus.cc:482] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:22:09.108389 1865 raft_consensus.cc:3037] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:09.111711 1865 raft_consensus.cc:504] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } }
I20260620 20:22:09.112416 1865 leader_election.cc:288] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb [CANDIDATE]: Term 1 election: Requested vote from peers 73b9b1e003d24555b6efc26bc1637f24 (127.31.235.130:36423), 72bec57a25ec469c81857d193612ac0c (127.31.235.131:41165)
I20260620 20:22:09.112840 1782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b9bafee9c284905b752e53efd9943d1" candidate_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72bec57a25ec469c81857d193612ac0c"
I20260620 20:22:09.112879 1638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b9bafee9c284905b752e53efd9943d1" candidate_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73b9b1e003d24555b6efc26bc1637f24"
I20260620 20:22:09.113068 1782 raft_consensus.cc:3037] T 9b9bafee9c284905b752e53efd9943d1 P 72bec57a25ec469c81857d193612ac0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:09.113123 1638 raft_consensus.cc:3037] T 9b9bafee9c284905b752e53efd9943d1 P 73b9b1e003d24555b6efc26bc1637f24 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:09.115626 1782 raft_consensus.cc:2454] T 9b9bafee9c284905b752e53efd9943d1 P 72bec57a25ec469c81857d193612ac0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e44674c3cb9f4d009494d3bb4c54d0fb in term 1.
I20260620 20:22:09.116115 1430 leader_election.cc:302] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72bec57a25ec469c81857d193612ac0c, e44674c3cb9f4d009494d3bb4c54d0fb; no voters:
I20260620 20:22:09.116132 1638 raft_consensus.cc:2454] T 9b9bafee9c284905b752e53efd9943d1 P 73b9b1e003d24555b6efc26bc1637f24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e44674c3cb9f4d009494d3bb4c54d0fb in term 1.
I20260620 20:22:09.116425 1865 raft_consensus.cc:2781] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:09.116811 1865 raft_consensus.cc:686] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb [term 1 LEADER]: Becoming Leader. State: Replica: e44674c3cb9f4d009494d3bb4c54d0fb, State: Running, Role: LEADER
I20260620 20:22:09.117082 1865 consensus_queue.cc:237] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } }
I20260620 20:22:09.118685 1857 tablet_replica.cc:439] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb: TxnStatusTablet state changed. Reason: New leader e44674c3cb9f4d009494d3bb4c54d0fb. Latest consensus state: current_term: 1 leader_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } } }
I20260620 20:22:09.118959 1857 tablet_replica.cc:442] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:22:09.119586 1888 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 20:22:09.119827 1888 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 20:22:09.120379 1341 catalog_manager.cc:5495] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb reported cstate change: term changed from 0 to 1, leader changed from <none> to e44674c3cb9f4d009494d3bb4c54d0fb (127.31.235.129). New cstate: current_term: 1 leader_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } health_report { overall_health: UNKNOWN } } }
I20260620 20:22:09.163122 1841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:09.157832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60884 (local address 127.31.235.129:37061)
0620 20:22:09.158056 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:09.158062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:09.158294 (+ 232us) server_negotiation.cc:407] Connection header received
0620 20:22:09.158591 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:09.158595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:09.158667 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:09.158763 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:09.159128 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:09.159920 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:09.161212 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:09.161721 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:09.161871 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:09.162297 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:09.162427 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:09.162522 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:09.162687 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 20:22:09.162764 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 20:22:09.163159 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:09.157834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:37061 (local address 127.0.0.1:60884)
0620 20:22:09.158195 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:09.158211 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:09.158401 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:09.158789 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:09.158794 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:09.158807 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:22:09.158983 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:09.158991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:09.160055 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:09.160063 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:09.161046 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:09.161057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:09.161872 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:09.161878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:09.162063 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:09.162117 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:22:09.162548 (+ 431us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:22:09.162568 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 20:22:09.162666 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 20:22:09.162816 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 20:22:09.166913 1782 raft_consensus.cc:1261] T 9b9bafee9c284905b752e53efd9943d1 P 72bec57a25ec469c81857d193612ac0c [term 1 FOLLOWER]: Refusing update from remote peer e44674c3cb9f4d009494d3bb4c54d0fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:09.166961 1638 raft_consensus.cc:1261] T 9b9bafee9c284905b752e53efd9943d1 P 73b9b1e003d24555b6efc26bc1637f24 [term 1 FOLLOWER]: Refusing update from remote peer e44674c3cb9f4d009494d3bb4c54d0fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:09.167562 1857 consensus_queue.cc:1035] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:09.167899 1865 consensus_queue.cc:1035] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:09.170576 1854 tablet_replica.cc:439] T 9b9bafee9c284905b752e53efd9943d1 P 72bec57a25ec469c81857d193612ac0c: TxnStatusTablet state changed. Reason: New leader e44674c3cb9f4d009494d3bb4c54d0fb. Latest consensus state: current_term: 1 leader_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } } }
I20260620 20:22:09.170815 1854 tablet_replica.cc:442] T 9b9bafee9c284905b752e53efd9943d1 P 72bec57a25ec469c81857d193612ac0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:22:09.172586 1865 tablet_replica.cc:439] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } } }
I20260620 20:22:09.172871 1865 tablet_replica.cc:442] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:22:09.174433 1860 mvcc.cc:205] Tried to move back new op lower bound from 7299018461862293504 to 7299018461666127872. Current Snapshot: MvccSnapshot[applied={T|T < 7299018461862293504 or (T in {7299018461862293504})}]
I20260620 20:22:09.175514 1854 tablet_replica.cc:439] T 9b9bafee9c284905b752e53efd9943d1 P 72bec57a25ec469c81857d193612ac0c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } } }
I20260620 20:22:09.175741 1854 tablet_replica.cc:442] T 9b9bafee9c284905b752e53efd9943d1 P 72bec57a25ec469c81857d193612ac0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:22:09.182274 1852 tablet_replica.cc:439] T 9b9bafee9c284905b752e53efd9943d1 P 73b9b1e003d24555b6efc26bc1637f24: TxnStatusTablet state changed. Reason: New leader e44674c3cb9f4d009494d3bb4c54d0fb. Latest consensus state: current_term: 1 leader_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } } }
I20260620 20:22:09.182548 1852 tablet_replica.cc:442] T 9b9bafee9c284905b752e53efd9943d1 P 73b9b1e003d24555b6efc26bc1637f24: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:22:09.184182 1857 tablet_replica.cc:439] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } } }
I20260620 20:22:09.184444 1857 tablet_replica.cc:442] T 9b9bafee9c284905b752e53efd9943d1 P e44674c3cb9f4d009494d3bb4c54d0fb: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:22:09.186384 1852 tablet_replica.cc:439] T 9b9bafee9c284905b752e53efd9943d1 P 73b9b1e003d24555b6efc26bc1637f24: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e44674c3cb9f4d009494d3bb4c54d0fb" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37061 } } peers { permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 } } peers { permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 } } }
I20260620 20:22:09.186627 1852 tablet_replica.cc:442] T 9b9bafee9c284905b752e53efd9943d1 P 73b9b1e003d24555b6efc26bc1637f24: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:22:09.200680 1894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:09.189272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:37061 (local address 127.31.235.129:59619)
0620 20:22:09.193557 (+ 4285us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:09.193577 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:09.193722 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:09.194090 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:09.194095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:09.194108 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:22:09.194477 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:09.194489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:09.196015 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:09.196021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:09.197987 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:09.198002 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:09.199078 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:09.199085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:09.199285 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:09.199358 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:22:09.199825 (+ 467us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:22:09.199840 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 20:22:09.199939 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 20:22:09.200205 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4074,"thread_start_us":3923,"threads_started":1}
I20260620 20:22:09.200903 1841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:09.189255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:59619 (local address 127.31.235.129:37061)
0620 20:22:09.189462 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:09.189470 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:09.193669 (+ 4199us) server_negotiation.cc:407] Connection header received
0620 20:22:09.193871 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:09.193877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:09.193962 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:09.194050 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:09.194631 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:09.195856 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:09.198131 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:09.198847 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:09.198983 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:09.199547 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:09.199698 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:09.199782 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:09.200442 (+ 660us) server_negotiation.cc:299] Negotiation successful
0620 20:22:09.200530 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 20:22:09.212931 1638 raft_consensus.cc:1261] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 73b9b1e003d24555b6efc26bc1637f24 [term 1 FOLLOWER]: Refusing update from remote peer e44674c3cb9f4d009494d3bb4c54d0fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:09.213007 1782 raft_consensus.cc:1261] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 72bec57a25ec469c81857d193612ac0c [term 1 FOLLOWER]: Refusing update from remote peer e44674c3cb9f4d009494d3bb4c54d0fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:09.213538 1865 consensus_queue.cc:1035] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "73b9b1e003d24555b6efc26bc1637f24" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:09.214080 1865 consensus_queue.cc:1035] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "72bec57a25ec469c81857d193612ac0c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:09.302294 1342 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:09.302552 1342 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:09.305953 1342 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3fdc64dd9c544aafbccd0277b18972ce: Sending DeleteTablet for 3 replicas of tablet 2e40c0b7d4ac42e682adbcc9207e4fd9
I20260620 20:22:09.306912 1618 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e40c0b7d4ac42e682adbcc9207e4fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:09 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:57754
I20260620 20:22:09.306907 1761 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e40c0b7d4ac42e682adbcc9207e4fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:09 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:46506
I20260620 20:22:09.307199 1870 tablet_replica.cc:330] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 72bec57a25ec469c81857d193612ac0c: stopping tablet replica
I20260620 20:22:09.307204 1872 tablet_replica.cc:330] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 73b9b1e003d24555b6efc26bc1637f24: stopping tablet replica
I20260620 20:22:09.307407 1872 raft_consensus.cc:2229] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 73b9b1e003d24555b6efc26bc1637f24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:22:09.307399 1870 raft_consensus.cc:2229] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 72bec57a25ec469c81857d193612ac0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:22:09.307592 1870 raft_consensus.cc:2258] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 72bec57a25ec469c81857d193612ac0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:09.307602 1872 raft_consensus.cc:2258] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 73b9b1e003d24555b6efc26bc1637f24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:09.308946 1872 ts_tablet_manager.cc:1905] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 73b9b1e003d24555b6efc26bc1637f24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:09.308995 1870 ts_tablet_manager.cc:1905] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 72bec57a25ec469c81857d193612ac0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:09.316107 1870 ts_tablet_manager.cc:1918] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 72bec57a25ec469c81857d193612ac0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:22:09.316356 1870 log.cc:1196] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 72bec57a25ec469c81857d193612ac0c: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/2e40c0b7d4ac42e682adbcc9207e4fd9
I20260620 20:22:09.316476 1872 ts_tablet_manager.cc:1918] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 73b9b1e003d24555b6efc26bc1637f24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:22:09.316861 1870 ts_tablet_manager.cc:1939] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 72bec57a25ec469c81857d193612ac0c: Deleting consensus metadata
I20260620 20:22:09.317010 1478 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e40c0b7d4ac42e682adbcc9207e4fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:09 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:60876
I20260620 20:22:09.317324 1871 tablet_replica.cc:330] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb: stopping tablet replica
I20260620 20:22:09.317574 1871 raft_consensus.cc:2229] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:22:09.317926 1871 raft_consensus.cc:2258] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:09.318298 1327 catalog_manager.cc:4841] TS 72bec57a25ec469c81857d193612ac0c (127.31.235.131:41165): tablet 2e40c0b7d4ac42e682adbcc9207e4fd9 (table test-table [id=3abf491cb5594c98bb9d3d45377ede09]) successfully deleted
I20260620 20:22:09.318466 1872 log.cc:1196] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 73b9b1e003d24555b6efc26bc1637f24: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/2e40c0b7d4ac42e682adbcc9207e4fd9
I20260620 20:22:09.318897 1872 ts_tablet_manager.cc:1939] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P 73b9b1e003d24555b6efc26bc1637f24: Deleting consensus metadata
I20260620 20:22:09.319542 1871 ts_tablet_manager.cc:1905] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:09.319864 1329 catalog_manager.cc:4841] TS 73b9b1e003d24555b6efc26bc1637f24 (127.31.235.130:36423): tablet 2e40c0b7d4ac42e682adbcc9207e4fd9 (table test-table [id=3abf491cb5594c98bb9d3d45377ede09]) successfully deleted
I20260620 20:22:09.327164 1871 ts_tablet_manager.cc:1918] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:22:09.327375 1871 log.cc:1196] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/2e40c0b7d4ac42e682adbcc9207e4fd9
I20260620 20:22:09.327858 1871 ts_tablet_manager.cc:1939] T 2e40c0b7d4ac42e682adbcc9207e4fd9 P e44674c3cb9f4d009494d3bb4c54d0fb: Deleting consensus metadata
I20260620 20:22:09.328898 1327 catalog_manager.cc:4841] TS e44674c3cb9f4d009494d3bb4c54d0fb (127.31.235.129:37061): tablet 2e40c0b7d4ac42e682adbcc9207e4fd9 (table test-table [id=3abf491cb5594c98bb9d3d45377ede09]) successfully deleted
Jun 20 20:22:09 dist-test-slave-zzwf krb5kdc[1297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986928, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:22:09.341703 1841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:09.321269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60890 (local address 127.31.235.129:37061)
0620 20:22:09.321466 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:09.321472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:09.325045 (+ 3573us) server_negotiation.cc:407] Connection header received
0620 20:22:09.325245 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:09.325256 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:09.325359 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:09.325475 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:09.327254 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:09.328474 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:09.330355 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:09.330929 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:09.331028 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:09.335590 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:09.335603 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:09.335621 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:09.335657 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:09.338795 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:09.339485 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:09.339496 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:09.339504 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:09.339632 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:09.340073 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:09.340078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:09.340083 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:09.340440 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:09.340751 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:09.341086 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 20:22:09.341314 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 20:22:09.342403 1544 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60890
I20260620 20:22:09.359228 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:09.345309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.31.235.190:40921)
0620 20:22:09.345562 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:09.345572 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:09.345735 (+ 163us) server_negotiation.cc:407] Connection header received
0620 20:22:09.345891 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:09.345894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:09.345954 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:09.346033 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:09.347595 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:09.348435 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:09.350204 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:09.350731 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:09.350826 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:09.353671 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:09.353676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:09.353679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:09.353711 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:09.356310 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:09.357092 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:09.357096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:09.357098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:09.357218 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:09.357624 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:09.357627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:09.357629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:09.357909 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:09.358092 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:09.358646 (+ 554us) server_negotiation.cc:299] Negotiation successful
0620 20:22:09.358933 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260620 20:22:09.359815 1342 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48254
I20260620 20:22:09.362079 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 1413
I20260620 20:22:09.376394 1841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:09.363269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60904 (local address 127.31.235.129:37061)
0620 20:22:09.363450 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:09.363455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:09.363474 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:22:09.363628 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:09.363634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:09.363710 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:09.363820 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:09.364932 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:09.366093 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:09.367339 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:09.368031 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:09.368168 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:09.370349 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:09.370356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:09.370361 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:09.370400 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:09.373876 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:09.374485 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:09.374491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:09.374495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:09.374672 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:09.375006 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:09.375012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:09.375015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:09.375357 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:09.375629 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:09.375789 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 20:22:09.376023 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 20:22:09.377099 1544 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60904
W20260620 20:22:09.377717 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 1413: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:22:09.377920 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 1413
I20260620 20:22:09.400148 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 1553
Jun 20 20:22:09 dist-test-slave-zzwf krb5kdc[1297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986928, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.235.130@KRBTEST.COM
I20260620 20:22:09.415536 1905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:09.401055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57758 (local address 127.31.235.130:36423)
0620 20:22:09.401630 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:09.401634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:09.401651 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:22:09.401708 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:09.401711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:09.401771 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:09.401943 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:09.403085 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:09.403808 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:09.404983 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:09.405379 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:09.405469 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:09.409728 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:09.409743 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:09.409760 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:09.409797 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:09.412693 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:09.413367 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:09.413374 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:09.413377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:09.413596 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:09.413987 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:09.413991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:09.413993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:09.414397 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:09.414731 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:09.414965 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 20:22:09.415251 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":288,"threads_started":1}
W20260620 20:22:09.416220 1683 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57758
W20260620 20:22:09.416779 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 1553: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:22:09.416925 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 1553
I20260620 20:22:09.433653 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 1692
Jun 20 20:22:09 dist-test-slave-zzwf krb5kdc[1297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986928, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.235.131@KRBTEST.COM
I20260620 20:22:09.449090 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:09.434366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46522 (local address 127.31.235.131:41165)
0620 20:22:09.434879 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:09.434883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:09.434899 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:22:09.434954 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:09.434958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:09.435023 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:09.435126 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:09.436396 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:09.437174 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:09.438590 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:09.439025 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:09.439126 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:09.443364 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:09.443378 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:09.443394 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:09.443428 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:09.446306 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:09.447006 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:09.447013 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:09.447016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:09.447145 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:09.447527 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:09.447533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:09.447537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:09.447909 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:09.448251 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:09.448470 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 20:22:09.448698 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":335,"threads_started":1}
W20260620 20:22:09.449851 1822 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46522
W20260620 20:22:09.450433 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 1692: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:22:09.450603 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 1692
I20260620 20:22:09.467330 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 1310
I20260620 20:22:09.468530 1382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:09.581928 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 1310
2026-06-20T20:22:09Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4754 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:09 dist-test-slave-zzwf krb5kdc[1910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:09 dist-test-slave-zzwf krb5kdc[1910](info): set up 2 sockets
Jun 20 20:22:09 dist-test-slave-zzwf krb5kdc[1910](info): commencing operation
krb5kdc: starting...
W20260620 20:22:10.674161 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:22:10 dist-test-slave-zzwf krb5kdc[1910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:22:10.873282 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:44933
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:44933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:11.202569 1923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:11.202869 1923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:11.202957 1923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:11.210225 1923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:11.210337 1923 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 20:22:11.210383 1923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:11.210435 1923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:11.210485 1923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:11.210546 1923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:11.220631 1923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:44933
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:44933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:11.222020 1923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:11.223292 1923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:11.235533 1928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:11.235560 1929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:11.235846 1923 server_base.cc:1034] running on GCE node
W20260620 20:22:11.236043 1931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:11.236877 1923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:11.238294 1923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:11.239531 1923 hybrid_clock.cc:648] HybridClock initialized: now 1781986931239472 us; error 29 us; skew 500 ppm
I20260620 20:22:11.239846 1923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:11 dist-test-slave-zzwf krb5kdc[1910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986931, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:11.242965 1923 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:22:11.244445 1923 webserver.cc:460] Webserver started at http://127.31.235.190:34059/ using document root <none> and password file <none>
I20260620 20:22:11.245112 1923 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:11.245203 1923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:11.245450 1923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:11.248888 1923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "2a53b64f90ab43a1937ea040a4f28372"
format_stamp: "Formatted at 2026-06-20 20:22:11 on dist-test-slave-zzwf"
server_key: "070104bb4b1e93aa001e82268c6f3c59"
server_key_iv: "b412f73905102df68aead7064716748f"
server_key_version: "encryptionkey@0"
I20260620 20:22:11.249861 1923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "2a53b64f90ab43a1937ea040a4f28372"
format_stamp: "Formatted at 2026-06-20 20:22:11 on dist-test-slave-zzwf"
server_key: "070104bb4b1e93aa001e82268c6f3c59"
server_key_iv: "b412f73905102df68aead7064716748f"
server_key_version: "encryptionkey@0"
I20260620 20:22:11.257776 1923 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260620 20:22:11.263100 1938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:11.265130 1923 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 20:22:11.265339 1923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "2a53b64f90ab43a1937ea040a4f28372"
format_stamp: "Formatted at 2026-06-20 20:22:11 on dist-test-slave-zzwf"
server_key: "070104bb4b1e93aa001e82268c6f3c59"
server_key_iv: "b412f73905102df68aead7064716748f"
server_key_version: "encryptionkey@0"
I20260620 20:22:11.265630 1923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:11.300594 1923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:11.304610 1923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:11.304836 1923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:11.325616 1923 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:44933
I20260620 20:22:11.325623 1999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:44933 every 8 connection(s)
I20260620 20:22:11.326987 1923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:11.330704 2000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:11.332376 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 1923
I20260620 20:22:11.332566 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:22:11.333207 32686 external_mini_cluster.cc:1428] Setting key 2d2b2e916134b9802a34a80ca6451673
I20260620 20:22:11.342587 2000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372: Bootstrap starting.
I20260620 20:22:11.346438 2000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372: Neither blocks nor log segments found. Creating new log.
Jun 20 20:22:11 dist-test-slave-zzwf krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:11.347560 2000 log.cc:824] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:11.350733 2000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372: No bootstrap required, opened a new log
I20260620 20:22:11.355816 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:11.335217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36768 (local address 127.31.235.190:44933)
0620 20:22:11.336049 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:11.336061 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:11.336108 (+ 47us) server_negotiation.cc:407] Connection header received
0620 20:22:11.337742 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:11.337763 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:11.338123 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:11.338427 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:11.340106 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:11.341388 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:11.343149 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:11.343770 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:11.343892 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:11.348437 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:11.348451 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:11.348470 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:11.348513 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:11.351628 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:11.352462 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:11.352468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:11.352490 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:11.352672 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:11.353052 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:11.353056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:11.353059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:11.353682 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:11.353945 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:11.354364 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 20:22:11.354777 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":401,"threads_started":1}
I20260620 20:22:11.359203 2000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a53b64f90ab43a1937ea040a4f28372" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 44933 } }
I20260620 20:22:11.359606 2000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:11.359715 2000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a53b64f90ab43a1937ea040a4f28372, State: Initialized, Role: FOLLOWER
I20260620 20:22:11.360239 2000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a53b64f90ab43a1937ea040a4f28372" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 44933 } }
I20260620 20:22:11.360479 2000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:11.360636 2000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:11.360800 2000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:11.363320 2000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a53b64f90ab43a1937ea040a4f28372" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 44933 } }
I20260620 20:22:11.363868 2000 leader_election.cc:302] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a53b64f90ab43a1937ea040a4f28372; no voters:
I20260620 20:22:11.364490 2000 leader_election.cc:288] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:11.364638 2005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:11.365314 2005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372 [term 1 LEADER]: Becoming Leader. State: Replica: 2a53b64f90ab43a1937ea040a4f28372, State: Running, Role: LEADER
I20260620 20:22:11.366097 2005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a53b64f90ab43a1937ea040a4f28372" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 44933 } }
I20260620 20:22:11.366798 2000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:11.369114 2006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a53b64f90ab43a1937ea040a4f28372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a53b64f90ab43a1937ea040a4f28372" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 44933 } } }
I20260620 20:22:11.369520 2006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:11.370055 2007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a53b64f90ab43a1937ea040a4f28372. Latest consensus state: current_term: 1 leader_uuid: "2a53b64f90ab43a1937ea040a4f28372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a53b64f90ab43a1937ea040a4f28372" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 44933 } } }
I20260620 20:22:11.370383 2007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:11.371296 2012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:11.375677 2012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:11.384596 2012 catalog_manager.cc:1347] Generated new cluster ID: ed7ba6abb7bd430388200e39dbdd417f
I20260620 20:22:11.384744 2012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:11.392235 2012 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:11.393462 2012 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:11.403062 2012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372: Generated new TSK 0
I20260620 20:22:11.403919 2012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
I20260620 20:22:11.454115 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:44933
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:11.777446 2026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:11.777839 2026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:11.778003 2026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:11.785260 2026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:11.785367 2026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:11.785501 2026 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 20:22:11.785593 2026 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:22:11.785693 2026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:22:11.795575 2026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.235.190:44933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:11.796888 2026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:11.798262 2026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:11.811126 2034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:11.811049 2031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:11.811134 2032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:11.812408 2026 server_base.cc:1034] running on GCE node
I20260620 20:22:11.813126 2026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:11.814199 2026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:11.815446 2026 hybrid_clock.cc:648] HybridClock initialized: now 1781986931815423 us; error 68 us; skew 500 ppm
I20260620 20:22:11.815766 2026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:11 dist-test-slave-zzwf krb5kdc[1910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986931, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:11.819236 2026 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:22:11.820755 2026 webserver.cc:460] Webserver started at http://127.31.235.129:45893/ using document root <none> and password file <none>
I20260620 20:22:11.821552 2026 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:11.821697 2026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:11.822024 2026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:11.824929 2026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a"
format_stamp: "Formatted at 2026-06-20 20:22:11 on dist-test-slave-zzwf"
server_key: "17e10d137040acd25808eaf5914bd592"
server_key_iv: "954c3027be9c815398bbd51a62130ed0"
server_key_version: "encryptionkey@0"
I20260620 20:22:11.825779 2026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a"
format_stamp: "Formatted at 2026-06-20 20:22:11 on dist-test-slave-zzwf"
server_key: "17e10d137040acd25808eaf5914bd592"
server_key_iv: "954c3027be9c815398bbd51a62130ed0"
server_key_version: "encryptionkey@0"
I20260620 20:22:11.830888 2026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 20:22:11.834717 2041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:11.836141 2026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:22:11.836334 2026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a"
format_stamp: "Formatted at 2026-06-20 20:22:11 on dist-test-slave-zzwf"
server_key: "17e10d137040acd25808eaf5914bd592"
server_key_iv: "954c3027be9c815398bbd51a62130ed0"
server_key_version: "encryptionkey@0"
I20260620 20:22:11.836553 2026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:11.850659 2026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:11.855129 2026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:11.855427 2026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:11.858160 2026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:11.858274 2026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:11.858418 2026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:11.858592 2026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Jun 20 20:22:11 dist-test-slave-zzwf krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986931, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:11.884124 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:11.858661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:34693 (local address 127.31.235.190:44933)
0620 20:22:11.859493 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:11.859498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:11.860390 (+ 892us) server_negotiation.cc:407] Connection header received
0620 20:22:11.862719 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:11.862725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:11.862828 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:11.863080 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:11.865609 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:11.867146 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:11.870191 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:11.871087 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:11.871275 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:11.875343 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:11.875350 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:11.875354 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:11.875398 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:11.879589 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:11.880936 (+ 1347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:11.880944 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:11.880949 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:11.881119 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:11.881858 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:11.881866 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:11.881871 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:11.882269 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:11.882447 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:11.883550 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0620 20:22:11.883762 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":632,"threads_started":1}
I20260620 20:22:11.885232 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:11.859362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:44933 (local address 127.31.235.129:34693)
0620 20:22:11.860199 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:11.860241 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:11.862323 (+ 2082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:11.863391 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:11.863409 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:11.864434 (+ 1025us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:11.865336 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:11.865360 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:11.867607 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:11.867612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:11.870015 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:11.870028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:11.871403 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:11.871410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:11.871602 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:11.872572 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:11.872608 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:11.875067 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:11.879883 (+ 4816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:11.879894 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:11.880005 (+ 111us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:11.880720 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:11.881290 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:11.881302 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:11.881309 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:11.881664 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:11.882435 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:11.882446 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:11.883050 (+ 604us) client_negotiation.cc:769] Sending connection context
0620 20:22:11.883456 (+ 406us) client_negotiation.cc:240] Negotiation successful
0620 20:22:11.884016 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":374,"threads_started":1}
I20260620 20:22:11.907739 2102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:11.891982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:44933 (local address 127.31.235.129:43087)
0620 20:22:11.892677 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:11.892701 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:11.892826 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:11.893250 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:11.893253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:11.893769 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:11.894377 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:11.894393 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:11.895724 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:11.895729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:11.897301 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:11.897324 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:11.898555 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:11.898568 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:11.898855 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:11.899832 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:11.899865 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:11.901199 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:11.904857 (+ 3658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:11.904866 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:11.904870 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:11.905322 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:11.905790 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:11.905795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:11.905800 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:11.906023 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:11.906871 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:11.906875 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:11.907038 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 20:22:11.907156 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 20:22:11.907421 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":436,"threads_started":1}
I20260620 20:22:11.908115 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:11.891551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:43087 (local address 127.31.235.190:44933)
0620 20:22:11.891735 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:11.891741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:11.892804 (+ 1063us) server_negotiation.cc:407] Connection header received
0620 20:22:11.892989 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:11.892998 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:11.893104 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:11.893235 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:11.894544 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:11.895586 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:11.897478 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:11.898257 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:11.898437 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:11.901462 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:11.901468 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:11.901472 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:11.901553 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:11.904681 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:11.905472 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:11.905480 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:11.905484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:11.905661 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:11.906188 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:11.906197 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:11.906202 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:11.906705 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:11.906888 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:11.907301 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 20:22:11.907611 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:22:11.920914 2026 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:36535
I20260620 20:22:11.920931 2161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:36535 every 8 connection(s)
I20260620 20:22:11.922374 2026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:22:11.923821 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 2026
I20260620 20:22:11.924044 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:22:11.924862 32686 external_mini_cluster.cc:1428] Setting key 3dcb27395a6a86f87222c0dfbb61ffb8
I20260620 20:22:11.924887 2162 heartbeater.cc:347] Connected to a master server at 127.31.235.190:44933
I20260620 20:22:11.925122 2162 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:11.925886 2162 heartbeater.cc:511] Master 127.31.235.190:44933 requested a full tablet report, sending...
I20260620 20:22:11.928105 1954 ts_manager.cc:194] Registered new tserver with Master: 6bc4886fe4cd4b06aa4be3f7e4c1c31a (127.31.235.129:36535)
I20260620 20:22:11.929620 1954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:34693
WARNING: no policy specified for kudu/127.31.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
I20260620 20:22:11.970064 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:44933
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:12.303263 2166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:12.303544 2166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:12.303644 2166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:12.310861 2166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:12.310973 2166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:12.311059 2166 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 20:22:12.311111 2166 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:22:12.311172 2166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:22:12.321064 2166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.235.190:44933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:12.322430 2166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:12.323697 2166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:12.338709 2171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:12.338581 2172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:12.339237 2174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:12.339959 2166 server_base.cc:1034] running on GCE node
I20260620 20:22:12.340605 2166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:12.341689 2166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:12.342911 2166 hybrid_clock.cc:648] HybridClock initialized: now 1781986932342862 us; error 43 us; skew 500 ppm
I20260620 20:22:12.343240 2166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:12 dist-test-slave-zzwf krb5kdc[1910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986932, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:12.346449 2166 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:22:12.347900 2166 webserver.cc:460] Webserver started at http://127.31.235.130:43345/ using document root <none> and password file <none>
I20260620 20:22:12.348663 2166 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:12.348812 2166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:12.349135 2166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:12.352268 2166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "777d22cfe7b942d0b1b0f8a994ebf687"
format_stamp: "Formatted at 2026-06-20 20:22:12 on dist-test-slave-zzwf"
server_key: "f36cffba750b85f09237d7a5808a4bb6"
server_key_iv: "70ee744f9c708f392ec5eb3dda73cfe8"
server_key_version: "encryptionkey@0"
I20260620 20:22:12.353073 2166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "777d22cfe7b942d0b1b0f8a994ebf687"
format_stamp: "Formatted at 2026-06-20 20:22:12 on dist-test-slave-zzwf"
server_key: "f36cffba750b85f09237d7a5808a4bb6"
server_key_iv: "70ee744f9c708f392ec5eb3dda73cfe8"
server_key_version: "encryptionkey@0"
I20260620 20:22:12.358274 2166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 20:22:12.361742 2181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:12.363044 2166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:22:12.363206 2166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "777d22cfe7b942d0b1b0f8a994ebf687"
format_stamp: "Formatted at 2026-06-20 20:22:12 on dist-test-slave-zzwf"
server_key: "f36cffba750b85f09237d7a5808a4bb6"
server_key_iv: "70ee744f9c708f392ec5eb3dda73cfe8"
server_key_version: "encryptionkey@0"
I20260620 20:22:12.363404 2166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:12.396447 2166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:12.400668 2166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:12.400961 2166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:12.403743 2166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:12.403883 2166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:12.404008 2166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:12.404100 2166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:22:12 dist-test-slave-zzwf krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986932, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:12.430488 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:12.404388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:38207 (local address 127.31.235.190:44933)
0620 20:22:12.404569 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:12.404574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:12.406005 (+ 1431us) server_negotiation.cc:407] Connection header received
0620 20:22:12.408113 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:12.408121 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:12.408213 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:12.408332 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:12.411212 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:12.412047 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:12.415423 (+ 3376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:12.415882 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:12.415981 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:12.421463 (+ 5482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:12.421472 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:12.421477 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:12.421548 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:12.425673 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:12.426878 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:12.426892 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:12.426895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:12.427023 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:12.427708 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:12.427714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:12.427718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:12.428018 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:12.428140 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:12.429736 (+ 1596us) server_negotiation.cc:299] Negotiation successful
0620 20:22:12.430030 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:22:12.432024 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:12.404967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:44933 (local address 127.31.235.130:38207)
0620 20:22:12.405823 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:12.405876 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:12.407845 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:12.408630 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:12.408647 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:12.409710 (+ 1063us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:12.410923 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:12.410957 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:12.412269 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:12.412277 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:12.415237 (+ 2960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:12.415264 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:12.416139 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:12.416149 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:12.416448 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:12.417378 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:12.417410 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:12.421159 (+ 3749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:12.425846 (+ 4687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:12.425856 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:12.425882 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:12.426695 (+ 813us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:12.427194 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:12.427202 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:12.427208 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:12.427551 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:12.428162 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:12.428176 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:12.428825 (+ 649us) client_negotiation.cc:769] Sending connection context
0620 20:22:12.429272 (+ 447us) client_negotiation.cc:240] Negotiation successful
0620 20:22:12.430038 (+ 766us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":381,"threads_started":1}
I20260620 20:22:12.453292 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:12.438025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:37411 (local address 127.31.235.190:44933)
0620 20:22:12.438206 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:12.438211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:12.438923 (+ 712us) server_negotiation.cc:407] Connection header received
0620 20:22:12.439141 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:12.439147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:12.439250 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:12.439356 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:12.440673 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:12.441570 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:12.443904 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:12.444559 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:12.444740 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:12.447260 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:12.447266 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:12.447272 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:12.447316 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:12.450525 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:12.451221 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:12.451230 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:12.451234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:12.451376 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:12.451763 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:12.451770 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:12.451774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:12.452099 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:12.452271 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:12.452685 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 20:22:12.452911 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:22:12.453461 2268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:12.438169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:44933 (local address 127.31.235.130:37411)
0620 20:22:12.438788 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:12.438804 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:12.438976 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:12.439540 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:12.439545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:12.440213 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:12.440517 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:12.440526 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:12.441742 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:12.441748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:12.443731 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:12.443743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:12.444807 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:12.444816 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:12.444992 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:12.445694 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:12.445712 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:12.447052 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:12.450677 (+ 3625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:12.450682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:12.450687 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:12.451090 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:12.451492 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:12.451500 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:12.451503 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:12.451644 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:12.452232 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:12.452237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:12.452432 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 20:22:12.452583 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 20:22:12.452928 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":334,"threads_started":1}
I20260620 20:22:12.460853 2166 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:34217
I20260620 20:22:12.460923 2300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:34217 every 8 connection(s)
I20260620 20:22:12.462119 2166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:22:12.464251 2301 heartbeater.cc:347] Connected to a master server at 127.31.235.190:44933
I20260620 20:22:12.464444 2301 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:12.465018 2301 heartbeater.cc:511] Master 127.31.235.190:44933 requested a full tablet report, sending...
I20260620 20:22:12.466399 1954 ts_manager.cc:194] Registered new tserver with Master: 777d22cfe7b942d0b1b0f8a994ebf687 (127.31.235.130:34217)
I20260620 20:22:12.467471 1954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:38207
I20260620 20:22:12.470765 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 2166
I20260620 20:22:12.470945 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:22:12.471423 32686 external_mini_cluster.cc:1428] Setting key d946d5905f21afdab81dfd8faaa0619c
WARNING: no policy specified for kudu/127.31.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
I20260620 20:22:12.513603 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:44933
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:12.838339 2305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:12.838621 2305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:12.838722 2305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:12.845891 2305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:12.846001 2305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:12.846084 2305 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 20:22:12.846138 2305 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:22:12.846273 2305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:22:12.856048 2305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.235.190:44933
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:12.857307 2305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:12.858566 2305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:12.872285 2311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:12.872731 2310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:12.872787 2313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:12.873654 2305 server_base.cc:1034] running on GCE node
I20260620 20:22:12.874321 2305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:12.875300 2305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:12.876528 2305 hybrid_clock.cc:648] HybridClock initialized: now 1781986932876460 us; error 65 us; skew 500 ppm
I20260620 20:22:12.876856 2305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:12 dist-test-slave-zzwf krb5kdc[1910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986932, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:12.886058 2305 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:22:12.887760 2305 webserver.cc:460] Webserver started at http://127.31.235.131:41803/ using document root <none> and password file <none>
I20260620 20:22:12.888487 2305 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:12.888643 2305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:12.888962 2305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:12.891990 2305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "5df6f872d433441d909995caf387b602"
format_stamp: "Formatted at 2026-06-20 20:22:12 on dist-test-slave-zzwf"
server_key: "05434dba681e08770f5aa5dfc32d4cb6"
server_key_iv: "5b88fff751053bd7e69589bc7fc311ba"
server_key_version: "encryptionkey@0"
I20260620 20:22:12.892755 2305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "5df6f872d433441d909995caf387b602"
format_stamp: "Formatted at 2026-06-20 20:22:12 on dist-test-slave-zzwf"
server_key: "05434dba681e08770f5aa5dfc32d4cb6"
server_key_iv: "5b88fff751053bd7e69589bc7fc311ba"
server_key_version: "encryptionkey@0"
I20260620 20:22:12.898072 2305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:22:12.901706 2320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:12.903034 2305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:22:12.903234 2305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "5df6f872d433441d909995caf387b602"
format_stamp: "Formatted at 2026-06-20 20:22:12 on dist-test-slave-zzwf"
server_key: "05434dba681e08770f5aa5dfc32d4cb6"
server_key_iv: "5b88fff751053bd7e69589bc7fc311ba"
server_key_version: "encryptionkey@0"
I20260620 20:22:12.903434 2305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:12.932487 2162 heartbeater.cc:503] Master 127.31.235.190:44933 was elected leader, sending a full tablet report...
I20260620 20:22:12.938635 2305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:12.943249 2305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:12.943539 2305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:12.946286 2305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:12.946405 2305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:12.946532 2305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:12.946630 2305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:22:12 dist-test-slave-zzwf krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986932, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:12.971182 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:12.946788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:59257 (local address 127.31.235.190:44933)
0620 20:22:12.946977 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:12.946985 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:12.948405 (+ 1420us) server_negotiation.cc:407] Connection header received
0620 20:22:12.950582 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:12.950588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:12.950682 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:12.950798 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:12.953290 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:12.954730 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:12.957428 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:12.958157 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:12.958311 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:12.962896 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:12.962905 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:12.962912 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:12.962959 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:12.967054 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:12.968131 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:12.968137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:12.968140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:12.968307 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:12.968802 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:12.968808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:12.968812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:12.969096 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:12.969259 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:12.970541 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0620 20:22:12.970816 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 20:22:12.972389 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:12.947473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:44933 (local address 127.31.235.131:59257)
0620 20:22:12.948238 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:12.948279 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:12.950293 (+ 2014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:12.951097 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:12.951111 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:12.951921 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:12.952982 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:12.953030 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:12.954974 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:12.954981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:12.957239 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:12.957263 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:12.958472 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:12.958479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:12.958666 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:12.959481 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:12.959518 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:12.962578 (+ 3060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:12.967265 (+ 4687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:12.967279 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:12.967300 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:12.967970 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:12.968461 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:12.968466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:12.968470 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:12.968678 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:12.969233 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:12.969242 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:12.969979 (+ 737us) client_negotiation.cc:769] Sending connection context
0620 20:22:12.970378 (+ 399us) client_negotiation.cc:240] Negotiation successful
0620 20:22:12.970991 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":348,"threads_started":1}
I20260620 20:22:12.995319 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:12.979757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:44933 (local address 127.31.235.131:43517)
0620 20:22:12.980488 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:12.980508 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:12.980678 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:12.981023 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:12.981027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:12.981483 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:12.982170 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:12.982187 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:12.983937 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:12.983941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:12.984965 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:12.984976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:12.986170 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:12.986175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:12.986430 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:12.987224 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:12.987255 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:12.988405 (+ 1150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:12.992476 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:12.992482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:12.992486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:12.992902 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:12.993299 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:12.993304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:12.993308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:12.993448 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:12.994214 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:12.994218 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:12.994387 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 20:22:12.994595 (+ 208us) client_negotiation.cc:240] Negotiation successful
0620 20:22:12.994994 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":410,"threads_started":1}
I20260620 20:22:12.995581 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:12.979740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:43517 (local address 127.31.235.190:44933)
0620 20:22:12.979973 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:12.979982 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:12.980604 (+ 622us) server_negotiation.cc:407] Connection header received
0620 20:22:12.980816 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:12.980822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:12.980897 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:12.980998 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:12.982353 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:12.983781 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:12.985106 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:12.985971 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:12.986107 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:12.988609 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:12.988618 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:12.988627 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:12.988678 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:12.992330 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:12.993033 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:12.993045 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:12.993048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:12.993177 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:12.993598 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:12.993605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:12.993609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:12.993963 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:12.994160 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:12.994728 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 20:22:12.995047 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:22:13.006773 2305 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:46523
I20260620 20:22:13.006872 2439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:46523 every 8 connection(s)
I20260620 20:22:13.008191 2305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:22:13.010303 2440 heartbeater.cc:347] Connected to a master server at 127.31.235.190:44933
I20260620 20:22:13.010540 2440 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:13.011184 2440 heartbeater.cc:511] Master 127.31.235.190:44933 requested a full tablet report, sending...
I20260620 20:22:13.012585 1954 ts_manager.cc:194] Registered new tserver with Master: 5df6f872d433441d909995caf387b602 (127.31.235.131:46523)
I20260620 20:22:13.012825 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 2305
I20260620 20:22:13.012988 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:22:13.013599 32686 external_mini_cluster.cc:1428] Setting key 2f6967904234225d25708ff5e907669c
I20260620 20:22:13.013804 1954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:59257
I20260620 20:22:13.016892 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:22:13 dist-test-slave-zzwf krb5kdc[1910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:22:13 dist-test-slave-zzwf krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:13.063078 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.049327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36774 (local address 127.31.235.190:44933)
0620 20:22:13.049609 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:13.049616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:13.049822 (+ 206us) server_negotiation.cc:407] Connection header received
0620 20:22:13.050029 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:13.050037 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:13.050106 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:13.050212 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:13.051612 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.052329 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.053635 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.054064 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.054200 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.057964 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:13.057970 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:13.057973 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:13.058008 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:13.060703 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:13.061335 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:13.061339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:13.061341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:13.061452 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:13.061932 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:13.061935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:13.061937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:13.062204 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:13.062377 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:13.062647 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 20:22:13.062812 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 20:22:13.067374 1954 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:22:13.074457 1954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:22:13.096323 2454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.086712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52586 (local address 127.31.235.129:36535)
0620 20:22:13.087982 (+ 1270us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:13.087994 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:13.088026 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:22:13.088137 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:13.088146 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:13.088459 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:13.089271 (+ 812us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:13.089516 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.091004 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.093658 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.095171 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.095353 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.095513 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:13.095819 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 20:22:13.095997 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1079,"thread_start_us":930,"threads_started":1}
I20260620 20:22:13.096395 2453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.086473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:36535 (local address 127.0.0.1:52586)
0620 20:22:13.087156 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:13.087204 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:13.087422 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:13.088691 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:13.088699 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:13.088729 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:13.089078 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.089090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.091149 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.091155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.093421 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.093461 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.095370 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.095376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.095559 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.095608 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 20:22:13.095754 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 20:22:13.095941 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":27,"thread_start_us":260,"threads_started":1}
I20260620 20:22:13.097404 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.088225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.31.235.130:34217)
0620 20:22:13.088912 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:13.088920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:13.088948 (+ 28us) server_negotiation.cc:407] Connection header received
0620 20:22:13.089052 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:13.089060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:13.089268 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:13.090545 (+ 1277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:13.090711 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.092057 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.094841 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.096085 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.096230 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.096395 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:13.096985 (+ 590us) server_negotiation.cc:299] Negotiation successful
0620 20:22:13.097134 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":420,"threads_started":1}
I20260620 20:22:13.097510 2455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.087664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:34217 (local address 127.0.0.1:54190)
0620 20:22:13.088337 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:13.088355 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:13.088459 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:13.089468 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:13.089472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:13.089827 (+ 355us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:13.090346 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.090364 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.092215 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.092224 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.094663 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.094674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.096572 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.096579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.096777 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.096813 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 20:22:13.096935 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 20:22:13.097064 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":1,"thread_start_us":422,"threads_started":1}
I20260620 20:22:13.101035 2457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.088525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:46523 (local address 127.0.0.1:50130)
0620 20:22:13.092733 (+ 4208us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:13.092753 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:13.092886 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:13.093528 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:13.093539 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:13.093566 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:13.094022 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.094032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.095919 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.095924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.098542 (+ 2618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.098558 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.100305 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.100311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.100477 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.100519 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 20:22:13.100610 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 20:22:13.100747 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4084,"thread_start_us":3949,"threads_started":1}
I20260620 20:22:13.100538 2231 tablet_service.cc:1467] Processing CreateTablet for tablet 7988484f94af4dce91a5ae713a6175d2 (DEFAULT_TABLE table=test-table [id=1313df9cc53a42e9862eefc6525d5c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:13.100458 2096 tablet_service.cc:1467] Processing CreateTablet for tablet 7988484f94af4dce91a5ae713a6175d2 (DEFAULT_TABLE table=test-table [id=1313df9cc53a42e9862eefc6525d5c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:13.101337 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.089005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50130 (local address 127.31.235.131:46523)
0620 20:22:13.089675 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:13.089686 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:13.092865 (+ 3179us) server_negotiation.cc:407] Connection header received
0620 20:22:13.093059 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:13.093067 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:13.093283 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:13.093527 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:13.094266 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.095750 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.098733 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.100106 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.100291 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.100441 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:13.100701 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 20:22:13.100861 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":303,"threads_started":1}
I20260620 20:22:13.101902 2231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7988484f94af4dce91a5ae713a6175d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:13.101936 2096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7988484f94af4dce91a5ae713a6175d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:13.104979 2370 tablet_service.cc:1467] Processing CreateTablet for tablet 7988484f94af4dce91a5ae713a6175d2 (DEFAULT_TABLE table=test-table [id=1313df9cc53a42e9862eefc6525d5c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:13.106464 2370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7988484f94af4dce91a5ae713a6175d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:13.113368 2459 tablet_bootstrap.cc:492] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Bootstrap starting.
I20260620 20:22:13.115833 2460 tablet_bootstrap.cc:492] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687: Bootstrap starting.
I20260620 20:22:13.117331 2459 tablet_bootstrap.cc:654] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:13.118647 2459 log.cc:824] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:13.119208 2460 tablet_bootstrap.cc:654] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:13.120669 2460 log.cc:824] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:13.123894 2462 tablet_bootstrap.cc:492] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602: Bootstrap starting.
I20260620 20:22:13.125425 2460 tablet_bootstrap.cc:492] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687: No bootstrap required, opened a new log
I20260620 20:22:13.125772 2460 ts_tablet_manager.cc:1397] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260620 20:22:13.126984 2459 tablet_bootstrap.cc:492] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: No bootstrap required, opened a new log
I20260620 20:22:13.127240 2459 ts_tablet_manager.cc:1397] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260620 20:22:13.127628 2462 tablet_bootstrap.cc:654] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:13.128760 2462 log.cc:824] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:13.131999 2462 tablet_bootstrap.cc:492] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602: No bootstrap required, opened a new log
I20260620 20:22:13.132256 2462 ts_tablet_manager.cc:1397] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260620 20:22:13.134707 2459 raft_consensus.cc:348] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.135144 2459 raft_consensus.cc:374] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:13.135262 2459 raft_consensus.cc:729] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bc4886fe4cd4b06aa4be3f7e4c1c31a, State: Initialized, Role: FOLLOWER
I20260620 20:22:13.135916 2459 consensus_queue.cc:260] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.136304 2460 raft_consensus.cc:348] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.136746 2460 raft_consensus.cc:374] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:13.136849 2460 raft_consensus.cc:729] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 777d22cfe7b942d0b1b0f8a994ebf687, State: Initialized, Role: FOLLOWER
I20260620 20:22:13.137522 2460 consensus_queue.cc:260] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.138720 2301 heartbeater.cc:503] Master 127.31.235.190:44933 was elected leader, sending a full tablet report...
I20260620 20:22:13.141010 2460 ts_tablet_manager.cc:1428] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687: Time spent starting tablet: real 0.015s user 0.008s sys 0.004s
I20260620 20:22:13.142554 2459 ts_tablet_manager.cc:1428] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Time spent starting tablet: real 0.015s user 0.005s sys 0.010s
I20260620 20:22:13.142397 2462 raft_consensus.cc:348] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.142779 2462 raft_consensus.cc:374] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:13.142948 2462 raft_consensus.cc:729] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df6f872d433441d909995caf387b602, State: Initialized, Role: FOLLOWER
I20260620 20:22:13.143558 2462 consensus_queue.cc:260] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.144642 2440 heartbeater.cc:503] Master 127.31.235.190:44933 was elected leader, sending a full tablet report...
I20260620 20:22:13.145148 2462 ts_tablet_manager.cc:1428] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602: Time spent starting tablet: real 0.013s user 0.008s sys 0.006s
W20260620 20:22:13.175612 2163 tablet.cc:2369] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:22:13.177274 2465 raft_consensus.cc:482] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:22:13.177582 2465 raft_consensus.cc:504] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.179327 2465 leader_election.cc:288] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 777d22cfe7b942d0b1b0f8a994ebf687 (127.31.235.130:34217), 5df6f872d433441d909995caf387b602 (127.31.235.131:46523)
I20260620 20:22:13.186744 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.179724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:47029 (local address 127.31.235.130:34217)
0620 20:22:13.179961 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:13.179969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:13.180424 (+ 455us) server_negotiation.cc:407] Connection header received
0620 20:22:13.180700 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:13.180708 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:13.180799 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:13.180895 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:13.181581 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.182513 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.184463 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.185622 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.185804 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.185883 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:13.186339 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 20:22:13.186465 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:22:13.186902 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.179670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:34217 (local address 127.31.235.129:47029)
0620 20:22:13.180306 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:13.180327 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:13.180542 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:13.180987 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:13.180991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:13.181015 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:13.181435 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.181448 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.182638 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.182645 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.184302 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.184313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.185978 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.185985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.186157 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.186196 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 20:22:13.186317 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 20:22:13.186461 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":1,"thread_start_us":300,"threads_started":1}
I20260620 20:22:13.187067 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.180667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:35975 (local address 127.31.235.131:46523)
0620 20:22:13.180889 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:13.180895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:13.181195 (+ 300us) server_negotiation.cc:407] Connection header received
0620 20:22:13.181350 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:13.181357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:13.181437 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:13.181570 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:13.182167 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.183113 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.185008 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.186128 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.186313 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.186385 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:13.186699 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 20:22:13.186818 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 20:22:13.187139 2469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.180342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:46523 (local address 127.31.235.129:35975)
0620 20:22:13.181079 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:13.181093 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:13.181220 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:13.181582 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:13.181586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:13.181615 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:13.182000 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.182011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.183257 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.183261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.184830 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.184843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.186342 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.186349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.186498 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.186521 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:22:13.186644 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 20:22:13.186772 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":456,"threads_started":1}
I20260620 20:22:13.187930 2251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7988484f94af4dce91a5ae713a6175d2" candidate_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" is_pre_election: true
I20260620 20:22:13.188129 2395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7988484f94af4dce91a5ae713a6175d2" candidate_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5df6f872d433441d909995caf387b602" is_pre_election: true
I20260620 20:22:13.188458 2251 raft_consensus.cc:2454] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bc4886fe4cd4b06aa4be3f7e4c1c31a in term 0.
I20260620 20:22:13.188665 2395 raft_consensus.cc:2454] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bc4886fe4cd4b06aa4be3f7e4c1c31a in term 0.
I20260620 20:22:13.189117 2044 leader_election.cc:302] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bc4886fe4cd4b06aa4be3f7e4c1c31a, 777d22cfe7b942d0b1b0f8a994ebf687; no voters:
I20260620 20:22:13.189540 2465 raft_consensus.cc:2781] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:22:13.189702 2465 raft_consensus.cc:482] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:22:13.189832 2465 raft_consensus.cc:3037] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:13.191936 2465 raft_consensus.cc:504] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.192517 2465 leader_election.cc:288] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [CANDIDATE]: Term 1 election: Requested vote from peers 777d22cfe7b942d0b1b0f8a994ebf687 (127.31.235.130:34217), 5df6f872d433441d909995caf387b602 (127.31.235.131:46523)
I20260620 20:22:13.192955 2251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7988484f94af4dce91a5ae713a6175d2" candidate_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "777d22cfe7b942d0b1b0f8a994ebf687"
I20260620 20:22:13.193161 2251 raft_consensus.cc:3037] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:13.193159 2395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7988484f94af4dce91a5ae713a6175d2" candidate_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5df6f872d433441d909995caf387b602"
I20260620 20:22:13.193394 2395 raft_consensus.cc:3037] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:13.195425 2251 raft_consensus.cc:2454] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bc4886fe4cd4b06aa4be3f7e4c1c31a in term 1.
I20260620 20:22:13.195590 2395 raft_consensus.cc:2454] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bc4886fe4cd4b06aa4be3f7e4c1c31a in term 1.
I20260620 20:22:13.195801 2044 leader_election.cc:302] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bc4886fe4cd4b06aa4be3f7e4c1c31a, 777d22cfe7b942d0b1b0f8a994ebf687; no voters:
I20260620 20:22:13.196094 2465 raft_consensus.cc:2781] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:13.196517 2465 raft_consensus.cc:686] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 1 LEADER]: Becoming Leader. State: Replica: 6bc4886fe4cd4b06aa4be3f7e4c1c31a, State: Running, Role: LEADER
I20260620 20:22:13.196980 2465 consensus_queue.cc:237] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.201229 1954 catalog_manager.cc:5495] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bc4886fe4cd4b06aa4be3f7e4c1c31a (127.31.235.129). New cstate: current_term: 1 leader_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } health_report { overall_health: UNKNOWN } } }
W20260620 20:22:13.214166 2302 tablet.cc:2369] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 20:22:13 dist-test-slave-zzwf krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986931, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:13.257121 2477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.244124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:44933 (local address 127.0.0.1:36780)
0620 20:22:13.244716 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:13.244737 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:13.244877 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:13.245198 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:13.245201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:13.245994 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:13.246324 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.246335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.247415 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.247418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.248431 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.248441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.249153 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.249157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.249277 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.249977 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:13.249995 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:13.252139 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:13.255004 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:13.255015 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:13.255021 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:13.255398 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:13.255706 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:13.255709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:13.255711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:13.255836 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:13.256297 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:13.256303 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:13.256467 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 20:22:13.256597 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 20:22:13.256796 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":333,"threads_started":1}
I20260620 20:22:13.257148 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.244025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36780 (local address 127.31.235.190:44933)
0620 20:22:13.244241 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:13.244247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:13.244824 (+ 577us) server_negotiation.cc:407] Connection header received
0620 20:22:13.245010 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:13.245014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:13.245077 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:13.245165 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:13.246493 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.247294 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.248568 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.248985 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.249092 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.252328 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:13.252333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:13.252337 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:13.252367 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:13.254876 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:13.255500 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:13.255504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:13.255506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:13.255617 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:13.255922 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:13.255926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:13.255927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:13.256197 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:13.256307 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:13.256696 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 20:22:13.256899 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":61}
W20260620 20:22:13.259752 2441 tablet.cc:2369] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:22:13.262310 1954 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:36780:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 20:22:13.263262 1954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:22:13.270138 2231 tablet_service.cc:1467] Processing CreateTablet for tablet 23d26a5cec3c4bca93c492433045e9b4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3392a4e603e4939819aababc4e59029]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:22:13.270355 2370 tablet_service.cc:1467] Processing CreateTablet for tablet 23d26a5cec3c4bca93c492433045e9b4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3392a4e603e4939819aababc4e59029]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:22:13.270138 2096 tablet_service.cc:1467] Processing CreateTablet for tablet 23d26a5cec3c4bca93c492433045e9b4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3392a4e603e4939819aababc4e59029]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:22:13.271111 2370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23d26a5cec3c4bca93c492433045e9b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:13.271111 2231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23d26a5cec3c4bca93c492433045e9b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:13.271124 2096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23d26a5cec3c4bca93c492433045e9b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:13.277386 2459 tablet_bootstrap.cc:492] T 23d26a5cec3c4bca93c492433045e9b4 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Bootstrap starting.
I20260620 20:22:13.278337 2460 tablet_bootstrap.cc:492] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687: Bootstrap starting.
I20260620 20:22:13.278806 2462 tablet_bootstrap.cc:492] T 23d26a5cec3c4bca93c492433045e9b4 P 5df6f872d433441d909995caf387b602: Bootstrap starting.
I20260620 20:22:13.279089 2459 tablet_bootstrap.cc:654] T 23d26a5cec3c4bca93c492433045e9b4 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:13.280156 2462 tablet_bootstrap.cc:654] T 23d26a5cec3c4bca93c492433045e9b4 P 5df6f872d433441d909995caf387b602: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:13.280254 2460 tablet_bootstrap.cc:654] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:13.281435 2459 tablet_bootstrap.cc:492] T 23d26a5cec3c4bca93c492433045e9b4 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: No bootstrap required, opened a new log
I20260620 20:22:13.281641 2459 ts_tablet_manager.cc:1397] T 23d26a5cec3c4bca93c492433045e9b4 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260620 20:22:13.282405 2459 raft_consensus.cc:348] T 23d26a5cec3c4bca93c492433045e9b4 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.282656 2459 raft_consensus.cc:374] T 23d26a5cec3c4bca93c492433045e9b4 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:13.282701 2460 tablet_bootstrap.cc:492] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687: No bootstrap required, opened a new log
I20260620 20:22:13.282788 2459 raft_consensus.cc:729] T 23d26a5cec3c4bca93c492433045e9b4 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bc4886fe4cd4b06aa4be3f7e4c1c31a, State: Initialized, Role: FOLLOWER
I20260620 20:22:13.282848 2460 ts_tablet_manager.cc:1397] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260620 20:22:13.283074 2459 consensus_queue.cc:260] T 23d26a5cec3c4bca93c492433045e9b4 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.283542 2465 tablet_replica.cc:439] T 23d26a5cec3c4bca93c492433045e9b4 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } } }
I20260620 20:22:13.283725 2459 ts_tablet_manager.cc:1428] T 23d26a5cec3c4bca93c492433045e9b4 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:22:13.283588 2460 raft_consensus.cc:348] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.283844 2465 tablet_replica.cc:442] T 23d26a5cec3c4bca93c492433045e9b4 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:22:13.283825 2460 raft_consensus.cc:374] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:13.283941 2460 raft_consensus.cc:729] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 777d22cfe7b942d0b1b0f8a994ebf687, State: Initialized, Role: FOLLOWER
I20260620 20:22:13.284192 2460 consensus_queue.cc:260] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.284894 2460 ts_tablet_manager.cc:1428] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 20:22:13.284725 2466 tablet_replica.cc:439] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } } }
I20260620 20:22:13.285111 2466 tablet_replica.cc:442] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:22:13.286502 2462 tablet_bootstrap.cc:492] T 23d26a5cec3c4bca93c492433045e9b4 P 5df6f872d433441d909995caf387b602: No bootstrap required, opened a new log
I20260620 20:22:13.286653 2462 ts_tablet_manager.cc:1397] T 23d26a5cec3c4bca93c492433045e9b4 P 5df6f872d433441d909995caf387b602: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260620 20:22:13.287487 2462 raft_consensus.cc:348] T 23d26a5cec3c4bca93c492433045e9b4 P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.287725 2462 raft_consensus.cc:374] T 23d26a5cec3c4bca93c492433045e9b4 P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:13.287874 2462 raft_consensus.cc:729] T 23d26a5cec3c4bca93c492433045e9b4 P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df6f872d433441d909995caf387b602, State: Initialized, Role: FOLLOWER
I20260620 20:22:13.288120 2462 consensus_queue.cc:260] T 23d26a5cec3c4bca93c492433045e9b4 P 5df6f872d433441d909995caf387b602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.288602 2462 ts_tablet_manager.cc:1428] T 23d26a5cec3c4bca93c492433045e9b4 P 5df6f872d433441d909995caf387b602: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 20:22:13.288549 2467 tablet_replica.cc:439] T 23d26a5cec3c4bca93c492433045e9b4 P 5df6f872d433441d909995caf387b602: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } } }
I20260620 20:22:13.288940 2467 tablet_replica.cc:442] T 23d26a5cec3c4bca93c492433045e9b4 P 5df6f872d433441d909995caf387b602: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:22:13.305684 2466 raft_consensus.cc:482] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:22:13.305895 2466 raft_consensus.cc:504] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.307617 2466 leader_election.cc:288] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bc4886fe4cd4b06aa4be3f7e4c1c31a (127.31.235.129:36535), 5df6f872d433441d909995caf387b602 (127.31.235.131:46523)
I20260620 20:22:13.314983 2454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.308037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:53807 (local address 127.31.235.129:36535)
0620 20:22:13.308280 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:13.308289 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:13.308685 (+ 396us) server_negotiation.cc:407] Connection header received
0620 20:22:13.308925 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:13.308933 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:13.309032 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:13.309128 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:13.309827 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.310836 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.312830 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.313989 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.314165 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.314239 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:13.314559 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 20:22:13.314742 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:22:13.315040 2481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.307976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:36535 (local address 127.31.235.130:53807)
0620 20:22:13.308573 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:13.308589 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:13.308759 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:13.309235 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:13.309240 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:13.309273 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:13.309663 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.309674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.310960 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.310968 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.312670 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.312681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.314219 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.314223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.314373 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.314406 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 20:22:13.314525 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 20:22:13.314655 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":339,"threads_started":1}
I20260620 20:22:13.315253 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.309000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:42871 (local address 127.31.235.131:46523)
0620 20:22:13.309176 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:13.309184 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:13.309439 (+ 255us) server_negotiation.cc:407] Connection header received
0620 20:22:13.309631 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:13.309636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:13.309712 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:13.309807 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:13.310412 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.311410 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.313147 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.314373 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.314506 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.314572 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:13.314877 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 20:22:13.314995 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:22:13.315886 2482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.308756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:46523 (local address 127.31.235.130:42871)
0620 20:22:13.309335 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:13.309358 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:13.309455 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:13.309827 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:13.309832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:13.309873 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:13.310241 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.310253 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.311610 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.311618 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.313004 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.313016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.314559 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.314565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.314709 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.314739 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 20:22:13.314835 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 20:22:13.314942 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":4,"thread_start_us":263,"threads_started":1}
I20260620 20:22:13.316057 2117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23d26a5cec3c4bca93c492433045e9b4" candidate_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" is_pre_election: true
I20260620 20:22:13.316464 2395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23d26a5cec3c4bca93c492433045e9b4" candidate_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5df6f872d433441d909995caf387b602" is_pre_election: true
I20260620 20:22:13.316469 2117 raft_consensus.cc:2454] T 23d26a5cec3c4bca93c492433045e9b4 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 777d22cfe7b942d0b1b0f8a994ebf687 in term 0.
I20260620 20:22:13.316690 2395 raft_consensus.cc:2454] T 23d26a5cec3c4bca93c492433045e9b4 P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 777d22cfe7b942d0b1b0f8a994ebf687 in term 0.
I20260620 20:22:13.317011 2184 leader_election.cc:302] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bc4886fe4cd4b06aa4be3f7e4c1c31a, 777d22cfe7b942d0b1b0f8a994ebf687; no voters:
I20260620 20:22:13.317399 2466 raft_consensus.cc:2781] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:22:13.317548 2466 raft_consensus.cc:482] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:22:13.317651 2466 raft_consensus.cc:3037] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:13.319864 2466 raft_consensus.cc:504] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.320485 2466 leader_election.cc:288] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687 [CANDIDATE]: Term 1 election: Requested vote from peers 6bc4886fe4cd4b06aa4be3f7e4c1c31a (127.31.235.129:36535), 5df6f872d433441d909995caf387b602 (127.31.235.131:46523)
I20260620 20:22:13.320960 2117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23d26a5cec3c4bca93c492433045e9b4" candidate_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a"
I20260620 20:22:13.321008 2395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23d26a5cec3c4bca93c492433045e9b4" candidate_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5df6f872d433441d909995caf387b602"
I20260620 20:22:13.321138 2117 raft_consensus.cc:3037] T 23d26a5cec3c4bca93c492433045e9b4 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:13.321239 2395 raft_consensus.cc:3037] T 23d26a5cec3c4bca93c492433045e9b4 P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:13.323225 2395 raft_consensus.cc:2454] T 23d26a5cec3c4bca93c492433045e9b4 P 5df6f872d433441d909995caf387b602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 777d22cfe7b942d0b1b0f8a994ebf687 in term 1.
I20260620 20:22:13.323246 2117 raft_consensus.cc:2454] T 23d26a5cec3c4bca93c492433045e9b4 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 777d22cfe7b942d0b1b0f8a994ebf687 in term 1.
I20260620 20:22:13.323587 2184 leader_election.cc:302] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bc4886fe4cd4b06aa4be3f7e4c1c31a, 777d22cfe7b942d0b1b0f8a994ebf687; no voters:
I20260620 20:22:13.323863 2466 raft_consensus.cc:2781] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:13.324288 2466 raft_consensus.cc:686] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 1 LEADER]: Becoming Leader. State: Replica: 777d22cfe7b942d0b1b0f8a994ebf687, State: Running, Role: LEADER
I20260620 20:22:13.324783 2466 consensus_queue.cc:237] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.328388 2483 tablet_replica.cc:439] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687: TxnStatusTablet state changed. Reason: New leader 777d22cfe7b942d0b1b0f8a994ebf687. Latest consensus state: current_term: 1 leader_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } } }
I20260620 20:22:13.328817 2483 tablet_replica.cc:442] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:22:13.329588 2485 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 20:22:13.329838 2485 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 20:22:13.330096 1954 catalog_manager.cc:5495] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687 reported cstate change: term changed from 0 to 1, leader changed from <none> to 777d22cfe7b942d0b1b0f8a994ebf687 (127.31.235.130). New cstate: current_term: 1 leader_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } health_report { overall_health: UNKNOWN } } }
I20260620 20:22:13.357409 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.351554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54200 (local address 127.31.235.130:34217)
0620 20:22:13.351735 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:13.351740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:13.351807 (+ 67us) server_negotiation.cc:407] Connection header received
0620 20:22:13.352072 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:13.352076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:13.352151 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:13.352273 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:13.352640 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.353430 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.355308 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.355715 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.355814 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.356349 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:13.356849 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:13.356951 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:13.357106 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 20:22:13.357170 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:22:13.357558 2477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.351448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:34217 (local address 127.0.0.1:54200)
0620 20:22:13.351712 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:13.351733 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:13.351889 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:13.352311 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:13.352316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:13.352333 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:22:13.352515 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.352526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.353572 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.353578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.355179 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.355193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.355883 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.355890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.356115 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.356184 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:22:13.356965 (+ 781us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:22:13.356989 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:22:13.357079 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 20:22:13.357230 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 20:22:13.365613 2117 raft_consensus.cc:1261] T 23d26a5cec3c4bca93c492433045e9b4 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 1 FOLLOWER]: Refusing update from remote peer 777d22cfe7b942d0b1b0f8a994ebf687: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:13.365816 2395 raft_consensus.cc:1261] T 23d26a5cec3c4bca93c492433045e9b4 P 5df6f872d433441d909995caf387b602 [term 1 FOLLOWER]: Refusing update from remote peer 777d22cfe7b942d0b1b0f8a994ebf687: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:13.366637 2483 consensus_queue.cc:1035] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:13.366962 2466 consensus_queue.cc:1035] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:13.379158 2465 tablet_replica.cc:439] T 23d26a5cec3c4bca93c492433045e9b4 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: TxnStatusTablet state changed. Reason: New leader 777d22cfe7b942d0b1b0f8a994ebf687. Latest consensus state: current_term: 1 leader_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } } }
I20260620 20:22:13.379500 2465 tablet_replica.cc:442] T 23d26a5cec3c4bca93c492433045e9b4 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:22:13.380139 2467 tablet_replica.cc:439] T 23d26a5cec3c4bca93c492433045e9b4 P 5df6f872d433441d909995caf387b602: TxnStatusTablet state changed. Reason: New leader 777d22cfe7b942d0b1b0f8a994ebf687. Latest consensus state: current_term: 1 leader_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } } }
I20260620 20:22:13.380424 2467 tablet_replica.cc:442] T 23d26a5cec3c4bca93c492433045e9b4 P 5df6f872d433441d909995caf387b602: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:22:13.384589 2483 tablet_replica.cc:439] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } } }
I20260620 20:22:13.384927 2483 tablet_replica.cc:442] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:22:13.389914 2493 tablet_replica.cc:439] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } } }
I20260620 20:22:13.390357 2493 tablet_replica.cc:442] T 23d26a5cec3c4bca93c492433045e9b4 P 777d22cfe7b942d0b1b0f8a994ebf687: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:22:13.410257 2467 tablet_replica.cc:439] T 23d26a5cec3c4bca93c492433045e9b4 P 5df6f872d433441d909995caf387b602: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } } }
I20260620 20:22:13.410605 2467 tablet_replica.cc:442] T 23d26a5cec3c4bca93c492433045e9b4 P 5df6f872d433441d909995caf387b602: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:22:13.413990 2465 tablet_replica.cc:439] T 23d26a5cec3c4bca93c492433045e9b4 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } } }
I20260620 20:22:13.414280 2465 tablet_replica.cc:442] T 23d26a5cec3c4bca93c492433045e9b4 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:22:13.416033 2454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.408229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.31.235.129:36535)
0620 20:22:13.408484 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:13.408493 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:13.408523 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:22:13.408672 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:13.408680 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:13.408825 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:13.409040 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:13.409726 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.410889 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.412933 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.413686 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.413808 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.414474 (+ 666us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:13.415179 (+ 705us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:13.415309 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:13.415535 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 20:22:13.415677 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:22:13.433008 2500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.426309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:34217 (local address 127.31.235.129:46583)
0620 20:22:13.427019 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:13.427036 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:13.427201 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:13.427905 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:13.427912 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:13.427929 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:22:13.428329 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.428342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.429860 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.429864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.430963 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.430974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.431957 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.431962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.432075 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.432127 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:22:13.432538 (+ 411us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:22:13.432546 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 20:22:13.432604 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 20:22:13.432737 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":378,"threads_started":1}
I20260620 20:22:13.433017 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.426340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:46583 (local address 127.31.235.130:34217)
0620 20:22:13.426572 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:13.426579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:13.427121 (+ 542us) server_negotiation.cc:407] Connection header received
0620 20:22:13.427432 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:13.427440 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:13.427534 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:13.427609 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:13.428491 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.429719 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.431103 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.431789 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.431919 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.432312 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:13.432445 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:13.432537 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:13.432648 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 20:22:13.432723 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 20:22:13.443840 2395 raft_consensus.cc:1261] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602 [term 1 FOLLOWER]: Refusing update from remote peer 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:13.443930 2251 raft_consensus.cc:1261] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 1 FOLLOWER]: Refusing update from remote peer 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:13.444656 2465 consensus_queue.cc:1035] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:13.445228 2465 consensus_queue.cc:1035] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [LEADER]: Connected to new peer: Peer: permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:13.450719 2490 mvcc.cc:205] Tried to move back new op lower bound from 7299018479378722816 to 7299018478378790912. Current Snapshot: MvccSnapshot[applied={T|T < 7299018479378722816}]
I20260620 20:22:13.451153 2489 mvcc.cc:205] Tried to move back new op lower bound from 7299018479378722816 to 7299018478378790912. Current Snapshot: MvccSnapshot[applied={T|T < 7299018479378722816}]
I20260620 20:22:13.489060 2454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.475503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:40843 (local address 127.31.235.129:36535)
0620 20:22:13.475684 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:13.475691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:13.482282 (+ 6591us) server_negotiation.cc:407] Connection header received
0620 20:22:13.482466 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:13.482473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:13.482565 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:13.482659 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:13.483440 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.484566 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.486799 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.487446 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.487547 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.488245 (+ 698us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:13.488381 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:13.488443 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:13.488671 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 20:22:13.488759 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:22:13.489908 2505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:13.475000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:36535 (local address 127.31.235.130:40843)
0620 20:22:13.482134 (+ 7134us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:13.482156 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:13.482322 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:13.482819 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:13.482825 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:13.482838 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:22:13.483223 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.483235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.484743 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.484750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.486636 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:13.486650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:13.487744 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:13.487751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:13.487958 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:13.488039 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:22:13.488519 (+ 480us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:22:13.488533 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 20:22:13.489314 (+ 781us) client_negotiation.cc:240] Negotiation successful
0620 20:22:13.489503 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6934,"thread_start_us":6769,"threads_started":1}
I20260620 20:22:13.534783 1954 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:13.535138 1954 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:13.539456 1954 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372: Sending DeleteTablet for 3 replicas of tablet 7988484f94af4dce91a5ae713a6175d2
I20260620 20:22:13.540649 2096 tablet_service.cc:1514] Processing DeleteTablet for tablet 7988484f94af4dce91a5ae713a6175d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:13 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:52586
I20260620 20:22:13.541185 2370 tablet_service.cc:1514] Processing DeleteTablet for tablet 7988484f94af4dce91a5ae713a6175d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:13 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:50130
I20260620 20:22:13.541939 2231 tablet_service.cc:1514] Processing DeleteTablet for tablet 7988484f94af4dce91a5ae713a6175d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:13 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:54190
I20260620 20:22:13.542999 2511 tablet_replica.cc:330] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687: stopping tablet replica
I20260620 20:22:13.543643 2511 raft_consensus.cc:2229] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:22:13.543766 1954 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 20:22:13.544417 2511 raft_consensus.cc:2258] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 20:22:13.544447 1954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:22:13.553815 2231 tablet_service.cc:1467] Processing CreateTablet for tablet af8a9e49567a47d78f0f3d180f8e058f (DEFAULT_TABLE table=test-table [id=883e322b7fdd4804a54ab74e4892c2c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:13.553828 2369 tablet_service.cc:1467] Processing CreateTablet for tablet af8a9e49567a47d78f0f3d180f8e058f (DEFAULT_TABLE table=test-table [id=883e322b7fdd4804a54ab74e4892c2c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:13.554296 2369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af8a9e49567a47d78f0f3d180f8e058f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:13.554312 2231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af8a9e49567a47d78f0f3d180f8e058f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:13.554195 2095 tablet_service.cc:1467] Processing CreateTablet for tablet af8a9e49567a47d78f0f3d180f8e058f (DEFAULT_TABLE table=test-table [id=883e322b7fdd4804a54ab74e4892c2c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:13.554744 2095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af8a9e49567a47d78f0f3d180f8e058f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:13.555433 2511 ts_tablet_manager.cc:1905] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:13.561514 2459 tablet_bootstrap.cc:492] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Bootstrap starting.
I20260620 20:22:13.561861 2460 tablet_bootstrap.cc:492] T af8a9e49567a47d78f0f3d180f8e058f P 777d22cfe7b942d0b1b0f8a994ebf687: Bootstrap starting.
I20260620 20:22:13.561898 2462 tablet_bootstrap.cc:492] T af8a9e49567a47d78f0f3d180f8e058f P 5df6f872d433441d909995caf387b602: Bootstrap starting.
I20260620 20:22:13.563202 2459 tablet_bootstrap.cc:654] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:13.563460 2460 tablet_bootstrap.cc:654] T af8a9e49567a47d78f0f3d180f8e058f P 777d22cfe7b942d0b1b0f8a994ebf687: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:13.563454 2462 tablet_bootstrap.cc:654] T af8a9e49567a47d78f0f3d180f8e058f P 5df6f872d433441d909995caf387b602: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:13.569561 2511 ts_tablet_manager.cc:1918] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:22:13.569931 2511 log.cc:1196] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/7988484f94af4dce91a5ae713a6175d2
I20260620 20:22:13.571992 2511 ts_tablet_manager.cc:1939] T 7988484f94af4dce91a5ae713a6175d2 P 777d22cfe7b942d0b1b0f8a994ebf687: Deleting consensus metadata
I20260620 20:22:13.574402 1941 catalog_manager.cc:4841] TS 777d22cfe7b942d0b1b0f8a994ebf687 (127.31.235.130:34217): tablet 7988484f94af4dce91a5ae713a6175d2 (table test-table [id=1313df9cc53a42e9862eefc6525d5c34]) successfully deleted
I20260620 20:22:13.576876 2509 tablet_replica.cc:330] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: stopping tablet replica
I20260620 20:22:13.578583 2460 tablet_bootstrap.cc:492] T af8a9e49567a47d78f0f3d180f8e058f P 777d22cfe7b942d0b1b0f8a994ebf687: No bootstrap required, opened a new log
I20260620 20:22:13.578879 2460 ts_tablet_manager.cc:1397] T af8a9e49567a47d78f0f3d180f8e058f P 777d22cfe7b942d0b1b0f8a994ebf687: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.008s
I20260620 20:22:13.579144 2510 tablet_replica.cc:330] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602: stopping tablet replica
I20260620 20:22:13.579583 2510 raft_consensus.cc:2229] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:22:13.579941 2510 raft_consensus.cc:2258] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:13.579895 2460 raft_consensus.cc:348] T af8a9e49567a47d78f0f3d180f8e058f P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.580225 2462 tablet_bootstrap.cc:492] T af8a9e49567a47d78f0f3d180f8e058f P 5df6f872d433441d909995caf387b602: No bootstrap required, opened a new log
I20260620 20:22:13.579120 2459 tablet_bootstrap.cc:492] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: No bootstrap required, opened a new log
I20260620 20:22:13.577463 2509 raft_consensus.cc:2229] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:22:13.580320 2460 raft_consensus.cc:374] T af8a9e49567a47d78f0f3d180f8e058f P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:13.580384 2462 ts_tablet_manager.cc:1397] T af8a9e49567a47d78f0f3d180f8e058f P 5df6f872d433441d909995caf387b602: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.005s
I20260620 20:22:13.580420 2459 ts_tablet_manager.cc:1397] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.001s
I20260620 20:22:13.580533 2460 raft_consensus.cc:729] T af8a9e49567a47d78f0f3d180f8e058f P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 777d22cfe7b942d0b1b0f8a994ebf687, State: Initialized, Role: FOLLOWER
I20260620 20:22:13.580843 2460 consensus_queue.cc:260] T af8a9e49567a47d78f0f3d180f8e058f P 777d22cfe7b942d0b1b0f8a994ebf687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.581039 2509 raft_consensus.cc:2258] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:13.581161 2462 raft_consensus.cc:348] T af8a9e49567a47d78f0f3d180f8e058f P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.581422 2462 raft_consensus.cc:374] T af8a9e49567a47d78f0f3d180f8e058f P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:13.581544 2462 raft_consensus.cc:729] T af8a9e49567a47d78f0f3d180f8e058f P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df6f872d433441d909995caf387b602, State: Initialized, Role: FOLLOWER
I20260620 20:22:13.581774 2462 consensus_queue.cc:260] T af8a9e49567a47d78f0f3d180f8e058f P 5df6f872d433441d909995caf387b602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.582172 2460 ts_tablet_manager.cc:1428] T af8a9e49567a47d78f0f3d180f8e058f P 777d22cfe7b942d0b1b0f8a994ebf687: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 20:22:13.582373 2462 ts_tablet_manager.cc:1428] T af8a9e49567a47d78f0f3d180f8e058f P 5df6f872d433441d909995caf387b602: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 20:22:13.581221 2459 raft_consensus.cc:348] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.582882 2459 raft_consensus.cc:374] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:13.582957 2459 raft_consensus.cc:729] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bc4886fe4cd4b06aa4be3f7e4c1c31a, State: Initialized, Role: FOLLOWER
I20260620 20:22:13.583199 2459 consensus_queue.cc:260] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.583770 2459 ts_tablet_manager.cc:1428] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260620 20:22:13.591063 2510 ts_tablet_manager.cc:1905] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:13.598773 2510 ts_tablet_manager.cc:1918] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:22:13.598958 2510 log.cc:1196] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/7988484f94af4dce91a5ae713a6175d2
I20260620 20:22:13.599500 2510 ts_tablet_manager.cc:1939] T 7988484f94af4dce91a5ae713a6175d2 P 5df6f872d433441d909995caf387b602: Deleting consensus metadata
I20260620 20:22:13.600409 2509 ts_tablet_manager.cc:1905] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:13.601364 1941 catalog_manager.cc:4841] TS 5df6f872d433441d909995caf387b602 (127.31.235.131:46523): tablet 7988484f94af4dce91a5ae713a6175d2 (table test-table [id=1313df9cc53a42e9862eefc6525d5c34]) successfully deleted
I20260620 20:22:13.608567 2509 ts_tablet_manager.cc:1918] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:22:13.608767 2509 log.cc:1196] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/7988484f94af4dce91a5ae713a6175d2
I20260620 20:22:13.609781 2509 ts_tablet_manager.cc:1939] T 7988484f94af4dce91a5ae713a6175d2 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Deleting consensus metadata
I20260620 20:22:13.611506 1941 catalog_manager.cc:4841] TS 6bc4886fe4cd4b06aa4be3f7e4c1c31a (127.31.235.129:36535): tablet 7988484f94af4dce91a5ae713a6175d2 (table test-table [id=1313df9cc53a42e9862eefc6525d5c34]) successfully deleted
I20260620 20:22:13.698191 2507 raft_consensus.cc:482] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:22:13.698449 2507 raft_consensus.cc:504] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.699256 2507 leader_election.cc:288] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 777d22cfe7b942d0b1b0f8a994ebf687 (127.31.235.130:34217), 5df6f872d433441d909995caf387b602 (127.31.235.131:46523)
I20260620 20:22:13.699874 2251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8a9e49567a47d78f0f3d180f8e058f" candidate_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" is_pre_election: true
I20260620 20:22:13.700004 2395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8a9e49567a47d78f0f3d180f8e058f" candidate_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5df6f872d433441d909995caf387b602" is_pre_election: true
I20260620 20:22:13.700153 2251 raft_consensus.cc:2454] T af8a9e49567a47d78f0f3d180f8e058f P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bc4886fe4cd4b06aa4be3f7e4c1c31a in term 0.
I20260620 20:22:13.700269 2395 raft_consensus.cc:2454] T af8a9e49567a47d78f0f3d180f8e058f P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bc4886fe4cd4b06aa4be3f7e4c1c31a in term 0.
I20260620 20:22:13.700809 2044 leader_election.cc:302] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bc4886fe4cd4b06aa4be3f7e4c1c31a, 777d22cfe7b942d0b1b0f8a994ebf687; no voters:
I20260620 20:22:13.701105 2507 raft_consensus.cc:2781] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:22:13.701248 2507 raft_consensus.cc:482] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:22:13.701332 2507 raft_consensus.cc:3037] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:13.704530 2507 raft_consensus.cc:504] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.705195 2507 leader_election.cc:288] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [CANDIDATE]: Term 1 election: Requested vote from peers 777d22cfe7b942d0b1b0f8a994ebf687 (127.31.235.130:34217), 5df6f872d433441d909995caf387b602 (127.31.235.131:46523)
I20260620 20:22:13.705832 2251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8a9e49567a47d78f0f3d180f8e058f" candidate_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "777d22cfe7b942d0b1b0f8a994ebf687"
I20260620 20:22:13.705966 2395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af8a9e49567a47d78f0f3d180f8e058f" candidate_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5df6f872d433441d909995caf387b602"
I20260620 20:22:13.706065 2251 raft_consensus.cc:3037] T af8a9e49567a47d78f0f3d180f8e058f P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:13.706184 2395 raft_consensus.cc:3037] T af8a9e49567a47d78f0f3d180f8e058f P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:13.709316 2395 raft_consensus.cc:2454] T af8a9e49567a47d78f0f3d180f8e058f P 5df6f872d433441d909995caf387b602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bc4886fe4cd4b06aa4be3f7e4c1c31a in term 1.
I20260620 20:22:13.709333 2251 raft_consensus.cc:2454] T af8a9e49567a47d78f0f3d180f8e058f P 777d22cfe7b942d0b1b0f8a994ebf687 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bc4886fe4cd4b06aa4be3f7e4c1c31a in term 1.
I20260620 20:22:13.709987 2044 leader_election.cc:302] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bc4886fe4cd4b06aa4be3f7e4c1c31a, 777d22cfe7b942d0b1b0f8a994ebf687; no voters:
I20260620 20:22:13.710367 2507 raft_consensus.cc:2781] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:13.710559 2507 raft_consensus.cc:686] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 1 LEADER]: Becoming Leader. State: Replica: 6bc4886fe4cd4b06aa4be3f7e4c1c31a, State: Running, Role: LEADER
I20260620 20:22:13.710870 2507 consensus_queue.cc:237] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:13.713881 1954 catalog_manager.cc:5495] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bc4886fe4cd4b06aa4be3f7e4c1c31a (127.31.235.129). New cstate: current_term: 1 leader_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } health_report { overall_health: UNKNOWN } } }
I20260620 20:22:13.781641 2395 raft_consensus.cc:1261] T af8a9e49567a47d78f0f3d180f8e058f P 5df6f872d433441d909995caf387b602 [term 1 FOLLOWER]: Refusing update from remote peer 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:13.782186 2470 consensus_queue.cc:1035] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:13.782785 2251 raft_consensus.cc:1261] T af8a9e49567a47d78f0f3d180f8e058f P 777d22cfe7b942d0b1b0f8a994ebf687 [term 1 FOLLOWER]: Refusing update from remote peer 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:13.783411 2470 consensus_queue.cc:1035] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [LEADER]: Connected to new peer: Peer: permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 20:22:13.825687 2490 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 20:22:15.417080 1955 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:15.417285 1955 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:15.421115 1955 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372: Sending DeleteTablet for 3 replicas of tablet af8a9e49567a47d78f0f3d180f8e058f
I20260620 20:22:15.422086 2096 tablet_service.cc:1514] Processing DeleteTablet for tablet af8a9e49567a47d78f0f3d180f8e058f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:15 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:52586
I20260620 20:22:15.422107 2231 tablet_service.cc:1514] Processing DeleteTablet for tablet af8a9e49567a47d78f0f3d180f8e058f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:15 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:54190
I20260620 20:22:15.422114 2370 tablet_service.cc:1514] Processing DeleteTablet for tablet af8a9e49567a47d78f0f3d180f8e058f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:15 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:50130
I20260620 20:22:15.434043 2542 tablet_replica.cc:330] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: stopping tablet replica
I20260620 20:22:15.434296 2542 raft_consensus.cc:2229] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:22:15.434671 2542 raft_consensus.cc:2258] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:15.435796 2542 ts_tablet_manager.cc:1905] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:15.440263 2541 tablet_replica.cc:330] T af8a9e49567a47d78f0f3d180f8e058f P 5df6f872d433441d909995caf387b602: stopping tablet replica
I20260620 20:22:15.440501 2541 raft_consensus.cc:2229] T af8a9e49567a47d78f0f3d180f8e058f P 5df6f872d433441d909995caf387b602 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:22:15.440843 2540 tablet_replica.cc:330] T af8a9e49567a47d78f0f3d180f8e058f P 777d22cfe7b942d0b1b0f8a994ebf687: stopping tablet replica
I20260620 20:22:15.440868 2541 raft_consensus.cc:2258] T af8a9e49567a47d78f0f3d180f8e058f P 5df6f872d433441d909995caf387b602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:15.441066 2540 raft_consensus.cc:2229] T af8a9e49567a47d78f0f3d180f8e058f P 777d22cfe7b942d0b1b0f8a994ebf687 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:22:15.441332 2540 raft_consensus.cc:2258] T af8a9e49567a47d78f0f3d180f8e058f P 777d22cfe7b942d0b1b0f8a994ebf687 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:15.442296 2541 ts_tablet_manager.cc:1905] T af8a9e49567a47d78f0f3d180f8e058f P 5df6f872d433441d909995caf387b602: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:15.442518 2540 ts_tablet_manager.cc:1905] T af8a9e49567a47d78f0f3d180f8e058f P 777d22cfe7b942d0b1b0f8a994ebf687: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:15.445644 2542 ts_tablet_manager.cc:1918] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:22:15.445834 2542 log.cc:1196] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/af8a9e49567a47d78f0f3d180f8e058f
I20260620 20:22:15.446322 2542 ts_tablet_manager.cc:1939] T af8a9e49567a47d78f0f3d180f8e058f P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Deleting consensus metadata
I20260620 20:22:15.447556 1941 catalog_manager.cc:4841] TS 6bc4886fe4cd4b06aa4be3f7e4c1c31a (127.31.235.129:36535): tablet af8a9e49567a47d78f0f3d180f8e058f (table test-table [id=883e322b7fdd4804a54ab74e4892c2c0]) successfully deleted
I20260620 20:22:15.448593 2541 ts_tablet_manager.cc:1918] T af8a9e49567a47d78f0f3d180f8e058f P 5df6f872d433441d909995caf387b602: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:22:15.448761 2541 log.cc:1196] T af8a9e49567a47d78f0f3d180f8e058f P 5df6f872d433441d909995caf387b602: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/af8a9e49567a47d78f0f3d180f8e058f
I20260620 20:22:15.449182 2541 ts_tablet_manager.cc:1939] T af8a9e49567a47d78f0f3d180f8e058f P 5df6f872d433441d909995caf387b602: Deleting consensus metadata
I20260620 20:22:15.449604 2540 ts_tablet_manager.cc:1918] T af8a9e49567a47d78f0f3d180f8e058f P 777d22cfe7b942d0b1b0f8a994ebf687: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:22:15.449739 2540 log.cc:1196] T af8a9e49567a47d78f0f3d180f8e058f P 777d22cfe7b942d0b1b0f8a994ebf687: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/af8a9e49567a47d78f0f3d180f8e058f
I20260620 20:22:15.450193 2540 ts_tablet_manager.cc:1939] T af8a9e49567a47d78f0f3d180f8e058f P 777d22cfe7b942d0b1b0f8a994ebf687: Deleting consensus metadata
I20260620 20:22:15.450323 1941 catalog_manager.cc:4841] TS 5df6f872d433441d909995caf387b602 (127.31.235.131:46523): tablet af8a9e49567a47d78f0f3d180f8e058f (table test-table [id=883e322b7fdd4804a54ab74e4892c2c0]) successfully deleted
I20260620 20:22:15.451128 1941 catalog_manager.cc:4841] TS 777d22cfe7b942d0b1b0f8a994ebf687 (127.31.235.130:34217): tablet af8a9e49567a47d78f0f3d180f8e058f (table test-table [id=883e322b7fdd4804a54ab74e4892c2c0]) successfully deleted
Jun 20 20:22:15 dist-test-slave-zzwf krb5kdc[1910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986935, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:22:15 dist-test-slave-zzwf krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986935, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:15.482622 2550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:15.467194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36782 (local address 127.31.235.190:44933)
0620 20:22:15.468125 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:15.468133 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:15.468158 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:22:15.468220 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:15.468225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:15.468302 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:15.468432 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:15.469908 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:15.470826 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:15.472126 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:15.472543 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:15.472675 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:15.477059 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:15.477065 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:15.477068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:15.477102 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:15.479961 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:15.480632 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:15.480638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:15.480642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:15.480768 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:15.481149 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:15.481156 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:15.481159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:15.481454 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:15.481765 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:15.482065 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 20:22:15.482276 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":616,"threads_started":1}
I20260620 20:22:15.485893 1955 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:22:15.486488 1955 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:22:15.492769 2096 tablet_service.cc:1467] Processing CreateTablet for tablet 89e76124036945c29a253106c90de931 (DEFAULT_TABLE table=test-table [id=4f8ffe02751e4bcd9dfff6826e8a9599]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:15.492898 2231 tablet_service.cc:1467] Processing CreateTablet for tablet 89e76124036945c29a253106c90de931 (DEFAULT_TABLE table=test-table [id=4f8ffe02751e4bcd9dfff6826e8a9599]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:15.493224 2096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89e76124036945c29a253106c90de931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:15.493264 2231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89e76124036945c29a253106c90de931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:15.493710 2370 tablet_service.cc:1467] Processing CreateTablet for tablet 89e76124036945c29a253106c90de931 (DEFAULT_TABLE table=test-table [id=4f8ffe02751e4bcd9dfff6826e8a9599]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:15.494086 2370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89e76124036945c29a253106c90de931. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:15.500103 2551 tablet_bootstrap.cc:492] T 89e76124036945c29a253106c90de931 P 5df6f872d433441d909995caf387b602: Bootstrap starting.
I20260620 20:22:15.501433 2551 tablet_bootstrap.cc:654] T 89e76124036945c29a253106c90de931 P 5df6f872d433441d909995caf387b602: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:15.506136 2553 tablet_bootstrap.cc:492] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Bootstrap starting.
I20260620 20:22:15.507055 2552 tablet_bootstrap.cc:492] T 89e76124036945c29a253106c90de931 P 777d22cfe7b942d0b1b0f8a994ebf687: Bootstrap starting.
I20260620 20:22:15.509354 2552 tablet_bootstrap.cc:654] T 89e76124036945c29a253106c90de931 P 777d22cfe7b942d0b1b0f8a994ebf687: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:15.510274 2551 tablet_bootstrap.cc:492] T 89e76124036945c29a253106c90de931 P 5df6f872d433441d909995caf387b602: No bootstrap required, opened a new log
I20260620 20:22:15.510545 2551 ts_tablet_manager.cc:1397] T 89e76124036945c29a253106c90de931 P 5df6f872d433441d909995caf387b602: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 20:22:15.511092 2553 tablet_bootstrap.cc:654] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:15.511320 2551 raft_consensus.cc:348] T 89e76124036945c29a253106c90de931 P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:15.511551 2552 tablet_bootstrap.cc:492] T 89e76124036945c29a253106c90de931 P 777d22cfe7b942d0b1b0f8a994ebf687: No bootstrap required, opened a new log
I20260620 20:22:15.511560 2551 raft_consensus.cc:374] T 89e76124036945c29a253106c90de931 P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:15.511683 2552 ts_tablet_manager.cc:1397] T 89e76124036945c29a253106c90de931 P 777d22cfe7b942d0b1b0f8a994ebf687: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 20:22:15.512012 2551 raft_consensus.cc:729] T 89e76124036945c29a253106c90de931 P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df6f872d433441d909995caf387b602, State: Initialized, Role: FOLLOWER
I20260620 20:22:15.512557 2551 consensus_queue.cc:260] T 89e76124036945c29a253106c90de931 P 5df6f872d433441d909995caf387b602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:15.512636 2552 raft_consensus.cc:348] T 89e76124036945c29a253106c90de931 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:15.512946 2552 raft_consensus.cc:374] T 89e76124036945c29a253106c90de931 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:15.513062 2552 raft_consensus.cc:729] T 89e76124036945c29a253106c90de931 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 777d22cfe7b942d0b1b0f8a994ebf687, State: Initialized, Role: FOLLOWER
I20260620 20:22:15.513548 2551 ts_tablet_manager.cc:1428] T 89e76124036945c29a253106c90de931 P 5df6f872d433441d909995caf387b602: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 20:22:15.513391 2552 consensus_queue.cc:260] T 89e76124036945c29a253106c90de931 P 777d22cfe7b942d0b1b0f8a994ebf687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:15.514300 2552 ts_tablet_manager.cc:1428] T 89e76124036945c29a253106c90de931 P 777d22cfe7b942d0b1b0f8a994ebf687: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 20:22:15.515823 2553 tablet_bootstrap.cc:492] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: No bootstrap required, opened a new log
I20260620 20:22:15.515988 2553 ts_tablet_manager.cc:1397] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 20:22:15.516783 2553 raft_consensus.cc:348] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:15.517031 2553 raft_consensus.cc:374] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:15.517128 2553 raft_consensus.cc:729] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bc4886fe4cd4b06aa4be3f7e4c1c31a, State: Initialized, Role: FOLLOWER
I20260620 20:22:15.517401 2553 consensus_queue.cc:260] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:15.518010 2553 ts_tablet_manager.cc:1428] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 20:22:15.618013 2533 raft_consensus.cc:482] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:22:15.618244 2533 raft_consensus.cc:504] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:15.618947 2533 leader_election.cc:288] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 777d22cfe7b942d0b1b0f8a994ebf687 (127.31.235.130:34217), 5df6f872d433441d909995caf387b602 (127.31.235.131:46523)
I20260620 20:22:15.619511 2250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89e76124036945c29a253106c90de931" candidate_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" is_pre_election: true
I20260620 20:22:15.619583 2394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89e76124036945c29a253106c90de931" candidate_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5df6f872d433441d909995caf387b602" is_pre_election: true
I20260620 20:22:15.619752 2250 raft_consensus.cc:2454] T 89e76124036945c29a253106c90de931 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bc4886fe4cd4b06aa4be3f7e4c1c31a in term 0.
I20260620 20:22:15.619906 2394 raft_consensus.cc:2454] T 89e76124036945c29a253106c90de931 P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bc4886fe4cd4b06aa4be3f7e4c1c31a in term 0.
I20260620 20:22:15.620249 2044 leader_election.cc:302] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bc4886fe4cd4b06aa4be3f7e4c1c31a, 777d22cfe7b942d0b1b0f8a994ebf687; no voters:
I20260620 20:22:15.620508 2533 raft_consensus.cc:2781] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:22:15.620611 2533 raft_consensus.cc:482] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:22:15.620719 2533 raft_consensus.cc:3037] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:15.622785 2533 raft_consensus.cc:504] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:15.623270 2533 leader_election.cc:288] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [CANDIDATE]: Term 1 election: Requested vote from peers 777d22cfe7b942d0b1b0f8a994ebf687 (127.31.235.130:34217), 5df6f872d433441d909995caf387b602 (127.31.235.131:46523)
I20260620 20:22:15.623855 2250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89e76124036945c29a253106c90de931" candidate_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "777d22cfe7b942d0b1b0f8a994ebf687"
I20260620 20:22:15.623855 2394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89e76124036945c29a253106c90de931" candidate_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5df6f872d433441d909995caf387b602"
I20260620 20:22:15.624064 2394 raft_consensus.cc:3037] T 89e76124036945c29a253106c90de931 P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:15.624065 2250 raft_consensus.cc:3037] T 89e76124036945c29a253106c90de931 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:15.627059 2250 raft_consensus.cc:2454] T 89e76124036945c29a253106c90de931 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bc4886fe4cd4b06aa4be3f7e4c1c31a in term 1.
I20260620 20:22:15.627064 2394 raft_consensus.cc:2454] T 89e76124036945c29a253106c90de931 P 5df6f872d433441d909995caf387b602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bc4886fe4cd4b06aa4be3f7e4c1c31a in term 1.
I20260620 20:22:15.627513 2044 leader_election.cc:302] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bc4886fe4cd4b06aa4be3f7e4c1c31a, 777d22cfe7b942d0b1b0f8a994ebf687; no voters:
I20260620 20:22:15.627800 2533 raft_consensus.cc:2781] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:15.627955 2533 raft_consensus.cc:686] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 1 LEADER]: Becoming Leader. State: Replica: 6bc4886fe4cd4b06aa4be3f7e4c1c31a, State: Running, Role: LEADER
I20260620 20:22:15.628242 2533 consensus_queue.cc:237] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:15.630772 1954 catalog_manager.cc:5495] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bc4886fe4cd4b06aa4be3f7e4c1c31a (127.31.235.129). New cstate: current_term: 1 leader_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } health_report { overall_health: UNKNOWN } } }
I20260620 20:22:15.666831 2560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:15.661546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.31.235.129:36535)
0620 20:22:15.662178 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:15.662183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:15.662201 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:22:15.662254 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:15.662258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:15.662315 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:15.662398 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:15.662749 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:15.663499 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:15.664831 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:15.665307 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:15.665421 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:15.665951 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:15.666058 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:15.666130 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:15.666297 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 20:22:15.666357 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":398,"threads_started":1}
I20260620 20:22:15.676446 2250 raft_consensus.cc:1261] T 89e76124036945c29a253106c90de931 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 1 FOLLOWER]: Refusing update from remote peer 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:15.676506 2394 raft_consensus.cc:1261] T 89e76124036945c29a253106c90de931 P 5df6f872d433441d909995caf387b602 [term 1 FOLLOWER]: Refusing update from remote peer 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:15.677004 2530 consensus_queue.cc:1035] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [LEADER]: Connected to new peer: Peer: permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:15.677321 2533 consensus_queue.cc:1035] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:15.683106 2519 mvcc.cc:205] Tried to move back new op lower bound from 7299018488525795328 to 7299018488335298560. Current Snapshot: MvccSnapshot[applied={T|T < 7299018488525795328}]
I20260620 20:22:15.723203 1953 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:15.723428 1953 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:15.726491 1953 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372: Sending DeleteTablet for 3 replicas of tablet 89e76124036945c29a253106c90de931
I20260620 20:22:15.727370 2096 tablet_service.cc:1514] Processing DeleteTablet for tablet 89e76124036945c29a253106c90de931 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:15 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:52586
I20260620 20:22:15.727452 2231 tablet_service.cc:1514] Processing DeleteTablet for tablet 89e76124036945c29a253106c90de931 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:15 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:54190
I20260620 20:22:15.727522 2370 tablet_service.cc:1514] Processing DeleteTablet for tablet 89e76124036945c29a253106c90de931 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:15 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:50130
I20260620 20:22:15.727751 2542 tablet_replica.cc:330] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: stopping tablet replica
I20260620 20:22:15.727782 2541 tablet_replica.cc:330] T 89e76124036945c29a253106c90de931 P 5df6f872d433441d909995caf387b602: stopping tablet replica
I20260620 20:22:15.727841 2540 tablet_replica.cc:330] T 89e76124036945c29a253106c90de931 P 777d22cfe7b942d0b1b0f8a994ebf687: stopping tablet replica
I20260620 20:22:15.727964 2541 raft_consensus.cc:2229] T 89e76124036945c29a253106c90de931 P 5df6f872d433441d909995caf387b602 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:22:15.727964 2542 raft_consensus.cc:2229] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:22:15.728041 2540 raft_consensus.cc:2229] T 89e76124036945c29a253106c90de931 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:22:15.728147 2541 raft_consensus.cc:2258] T 89e76124036945c29a253106c90de931 P 5df6f872d433441d909995caf387b602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:15.728214 2540 raft_consensus.cc:2258] T 89e76124036945c29a253106c90de931 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:15.728294 2542 raft_consensus.cc:2258] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:15.728462 1953 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:22:15.729115 1953 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:22:15.729430 2542 ts_tablet_manager.cc:1905] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:15.729528 2540 ts_tablet_manager.cc:1905] T 89e76124036945c29a253106c90de931 P 777d22cfe7b942d0b1b0f8a994ebf687: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:15.730211 2541 ts_tablet_manager.cc:1905] T 89e76124036945c29a253106c90de931 P 5df6f872d433441d909995caf387b602: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:15.736049 2540 ts_tablet_manager.cc:1918] T 89e76124036945c29a253106c90de931 P 777d22cfe7b942d0b1b0f8a994ebf687: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:22:15.736220 2540 log.cc:1196] T 89e76124036945c29a253106c90de931 P 777d22cfe7b942d0b1b0f8a994ebf687: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/89e76124036945c29a253106c90de931
I20260620 20:22:15.736670 2540 ts_tablet_manager.cc:1939] T 89e76124036945c29a253106c90de931 P 777d22cfe7b942d0b1b0f8a994ebf687: Deleting consensus metadata
I20260620 20:22:15.736857 2231 tablet_service.cc:1467] Processing CreateTablet for tablet e1b656691604454791305c19f0cbf844 (DEFAULT_TABLE table=test-table [id=f379f944108343ebad2dbb375a2a2712]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:15.736946 2370 tablet_service.cc:1467] Processing CreateTablet for tablet e1b656691604454791305c19f0cbf844 (DEFAULT_TABLE table=test-table [id=f379f944108343ebad2dbb375a2a2712]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:15.737278 2231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b656691604454791305c19f0cbf844. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:15.737322 2370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b656691604454791305c19f0cbf844. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:15.737470 2096 tablet_service.cc:1467] Processing CreateTablet for tablet e1b656691604454791305c19f0cbf844 (DEFAULT_TABLE table=test-table [id=f379f944108343ebad2dbb375a2a2712]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:15.737926 2096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1b656691604454791305c19f0cbf844. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:15.740911 2542 ts_tablet_manager.cc:1918] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:22:15.741475 2542 log.cc:1196] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/89e76124036945c29a253106c90de931
I20260620 20:22:15.742027 2542 ts_tablet_manager.cc:1939] T 89e76124036945c29a253106c90de931 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Deleting consensus metadata
I20260620 20:22:15.743033 2541 ts_tablet_manager.cc:1918] T 89e76124036945c29a253106c90de931 P 5df6f872d433441d909995caf387b602: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:22:15.743119 1941 catalog_manager.cc:4841] TS 6bc4886fe4cd4b06aa4be3f7e4c1c31a (127.31.235.129:36535): tablet 89e76124036945c29a253106c90de931 (table test-table [id=4f8ffe02751e4bcd9dfff6826e8a9599]) successfully deleted
I20260620 20:22:15.743201 2541 log.cc:1196] T 89e76124036945c29a253106c90de931 P 5df6f872d433441d909995caf387b602: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/89e76124036945c29a253106c90de931
I20260620 20:22:15.743577 2552 tablet_bootstrap.cc:492] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687: Bootstrap starting.
I20260620 20:22:15.743589 2541 ts_tablet_manager.cc:1939] T 89e76124036945c29a253106c90de931 P 5df6f872d433441d909995caf387b602: Deleting consensus metadata
I20260620 20:22:15.744102 2553 tablet_bootstrap.cc:492] T e1b656691604454791305c19f0cbf844 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Bootstrap starting.
I20260620 20:22:15.745036 1941 catalog_manager.cc:4841] TS 5df6f872d433441d909995caf387b602 (127.31.235.131:46523): tablet 89e76124036945c29a253106c90de931 (table test-table [id=4f8ffe02751e4bcd9dfff6826e8a9599]) successfully deleted
I20260620 20:22:15.745258 2552 tablet_bootstrap.cc:654] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:15.745285 1941 catalog_manager.cc:4841] TS 777d22cfe7b942d0b1b0f8a994ebf687 (127.31.235.130:34217): tablet 89e76124036945c29a253106c90de931 (table test-table [id=4f8ffe02751e4bcd9dfff6826e8a9599]) successfully deleted
I20260620 20:22:15.745707 2553 tablet_bootstrap.cc:654] T e1b656691604454791305c19f0cbf844 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:15.747416 2552 tablet_bootstrap.cc:492] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687: No bootstrap required, opened a new log
I20260620 20:22:15.747603 2552 ts_tablet_manager.cc:1397] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 20:22:15.747876 2551 tablet_bootstrap.cc:492] T e1b656691604454791305c19f0cbf844 P 5df6f872d433441d909995caf387b602: Bootstrap starting.
I20260620 20:22:15.748338 2552 raft_consensus.cc:348] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:15.748623 2552 raft_consensus.cc:374] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:15.748673 2553 tablet_bootstrap.cc:492] T e1b656691604454791305c19f0cbf844 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: No bootstrap required, opened a new log
I20260620 20:22:15.748704 2552 raft_consensus.cc:729] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 777d22cfe7b942d0b1b0f8a994ebf687, State: Initialized, Role: FOLLOWER
I20260620 20:22:15.748855 2553 ts_tablet_manager.cc:1397] T e1b656691604454791305c19f0cbf844 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260620 20:22:15.748940 2552 consensus_queue.cc:260] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:15.749541 2551 tablet_bootstrap.cc:654] T e1b656691604454791305c19f0cbf844 P 5df6f872d433441d909995caf387b602: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:15.749557 2552 ts_tablet_manager.cc:1428] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 20:22:15.749651 2553 raft_consensus.cc:348] T e1b656691604454791305c19f0cbf844 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:15.749995 2553 raft_consensus.cc:374] T e1b656691604454791305c19f0cbf844 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:15.750075 2553 raft_consensus.cc:729] T e1b656691604454791305c19f0cbf844 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bc4886fe4cd4b06aa4be3f7e4c1c31a, State: Initialized, Role: FOLLOWER
I20260620 20:22:15.750322 2553 consensus_queue.cc:260] T e1b656691604454791305c19f0cbf844 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:15.750852 2553 ts_tablet_manager.cc:1428] T e1b656691604454791305c19f0cbf844 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 20:22:15.758677 2551 tablet_bootstrap.cc:492] T e1b656691604454791305c19f0cbf844 P 5df6f872d433441d909995caf387b602: No bootstrap required, opened a new log
I20260620 20:22:15.758888 2551 ts_tablet_manager.cc:1397] T e1b656691604454791305c19f0cbf844 P 5df6f872d433441d909995caf387b602: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260620 20:22:15.759505 2551 raft_consensus.cc:348] T e1b656691604454791305c19f0cbf844 P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:15.759729 2551 raft_consensus.cc:374] T e1b656691604454791305c19f0cbf844 P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:15.759828 2551 raft_consensus.cc:729] T e1b656691604454791305c19f0cbf844 P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df6f872d433441d909995caf387b602, State: Initialized, Role: FOLLOWER
I20260620 20:22:15.760095 2551 consensus_queue.cc:260] T e1b656691604454791305c19f0cbf844 P 5df6f872d433441d909995caf387b602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:15.760555 2551 ts_tablet_manager.cc:1428] T e1b656691604454791305c19f0cbf844 P 5df6f872d433441d909995caf387b602: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 20:22:15.978793 2535 raft_consensus.cc:482] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:22:15.979101 2535 raft_consensus.cc:504] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:15.979811 2535 leader_election.cc:288] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bc4886fe4cd4b06aa4be3f7e4c1c31a (127.31.235.129:36535), 5df6f872d433441d909995caf387b602 (127.31.235.131:46523)
I20260620 20:22:15.980357 2117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b656691604454791305c19f0cbf844" candidate_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" is_pre_election: true
I20260620 20:22:15.980384 2395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b656691604454791305c19f0cbf844" candidate_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5df6f872d433441d909995caf387b602" is_pre_election: true
I20260620 20:22:15.980609 2117 raft_consensus.cc:2454] T e1b656691604454791305c19f0cbf844 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 777d22cfe7b942d0b1b0f8a994ebf687 in term 0.
I20260620 20:22:15.980608 2395 raft_consensus.cc:2454] T e1b656691604454791305c19f0cbf844 P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 777d22cfe7b942d0b1b0f8a994ebf687 in term 0.
I20260620 20:22:15.981077 2184 leader_election.cc:302] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5df6f872d433441d909995caf387b602, 777d22cfe7b942d0b1b0f8a994ebf687; no voters:
I20260620 20:22:15.981340 2535 raft_consensus.cc:2781] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:22:15.981443 2535 raft_consensus.cc:482] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:22:15.981573 2535 raft_consensus.cc:3037] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:15.983747 2535 raft_consensus.cc:504] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:15.984284 2535 leader_election.cc:288] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 [CANDIDATE]: Term 1 election: Requested vote from peers 6bc4886fe4cd4b06aa4be3f7e4c1c31a (127.31.235.129:36535), 5df6f872d433441d909995caf387b602 (127.31.235.131:46523)
I20260620 20:22:15.984834 2117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b656691604454791305c19f0cbf844" candidate_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a"
I20260620 20:22:15.984879 2395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1b656691604454791305c19f0cbf844" candidate_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5df6f872d433441d909995caf387b602"
I20260620 20:22:15.985059 2117 raft_consensus.cc:3037] T e1b656691604454791305c19f0cbf844 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:15.985075 2395 raft_consensus.cc:3037] T e1b656691604454791305c19f0cbf844 P 5df6f872d433441d909995caf387b602 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:15.987287 2117 raft_consensus.cc:2454] T e1b656691604454791305c19f0cbf844 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 777d22cfe7b942d0b1b0f8a994ebf687 in term 1.
I20260620 20:22:15.987352 2395 raft_consensus.cc:2454] T e1b656691604454791305c19f0cbf844 P 5df6f872d433441d909995caf387b602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 777d22cfe7b942d0b1b0f8a994ebf687 in term 1.
I20260620 20:22:15.987741 2184 leader_election.cc:302] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bc4886fe4cd4b06aa4be3f7e4c1c31a, 777d22cfe7b942d0b1b0f8a994ebf687; no voters:
I20260620 20:22:15.988090 2535 raft_consensus.cc:2781] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:15.988296 2535 raft_consensus.cc:686] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 1 LEADER]: Becoming Leader. State: Replica: 777d22cfe7b942d0b1b0f8a994ebf687, State: Running, Role: LEADER
I20260620 20:22:15.988560 2535 consensus_queue.cc:237] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } }
I20260620 20:22:15.991379 1953 catalog_manager.cc:5495] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 reported cstate change: term changed from 0 to 1, leader changed from <none> to 777d22cfe7b942d0b1b0f8a994ebf687 (127.31.235.130). New cstate: current_term: 1 leader_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "777d22cfe7b942d0b1b0f8a994ebf687" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 } health_report { overall_health: UNKNOWN } } }
I20260620 20:22:16.053345 2569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:16.048491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54204 (local address 127.31.235.130:34217)
0620 20:22:16.049070 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:16.049074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:16.049093 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:22:16.049139 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:16.049142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:16.049197 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:16.049276 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:16.049753 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:16.050534 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:16.051802 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:16.052193 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:16.052280 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:16.052633 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:16.052729 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:16.052793 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:16.052940 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 20:22:16.052996 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":318,"threads_started":1}
I20260620 20:22:16.065712 2569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:16.061089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:41633 (local address 127.31.235.130:34217)
0620 20:22:16.061323 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:16.061328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:16.061597 (+ 269us) server_negotiation.cc:407] Connection header received
0620 20:22:16.061729 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:16.061733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:16.061791 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:16.061900 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:16.062265 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:16.062939 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:16.064205 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:16.064603 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:16.064695 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:16.065063 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:16.065206 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:16.065272 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:16.065370 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 20:22:16.065432 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":62}
I20260620 20:22:16.065917 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:16.060972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:34217 (local address 127.31.235.130:41633)
0620 20:22:16.061514 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:16.061529 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:16.061630 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:16.061923 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:16.061926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:16.061934 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:22:16.062131 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:16.062138 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:16.063066 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:16.063069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:16.064079 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:16.064090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:16.064738 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:16.064742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:16.064900 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:16.064934 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:22:16.065287 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:22:16.065294 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 20:22:16.065347 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 20:22:16.065529 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":273,"threads_started":1}
I20260620 20:22:16.073995 2395 raft_consensus.cc:1261] T e1b656691604454791305c19f0cbf844 P 5df6f872d433441d909995caf387b602 [term 1 FOLLOWER]: Refusing update from remote peer 777d22cfe7b942d0b1b0f8a994ebf687: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:16.074083 2117 raft_consensus.cc:1261] T e1b656691604454791305c19f0cbf844 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 1 FOLLOWER]: Refusing update from remote peer 777d22cfe7b942d0b1b0f8a994ebf687: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:16.074632 2568 consensus_queue.cc:1035] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bc4886fe4cd4b06aa4be3f7e4c1c31a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:16.074985 2518 consensus_queue.cc:1035] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5df6f872d433441d909995caf387b602" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 20:22:16.094944 2519 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 20:22:17.703688 1952 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:17.703955 1952 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:17.708233 1952 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a53b64f90ab43a1937ea040a4f28372: Sending DeleteTablet for 3 replicas of tablet e1b656691604454791305c19f0cbf844
I20260620 20:22:17.709038 2096 tablet_service.cc:1514] Processing DeleteTablet for tablet e1b656691604454791305c19f0cbf844 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:17 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:52586
I20260620 20:22:17.709124 2370 tablet_service.cc:1514] Processing DeleteTablet for tablet e1b656691604454791305c19f0cbf844 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:17 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:50130
I20260620 20:22:17.709110 2231 tablet_service.cc:1514] Processing DeleteTablet for tablet e1b656691604454791305c19f0cbf844 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:17 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:54190
I20260620 20:22:17.709897 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 2026
I20260620 20:22:17.721654 2597 tablet_replica.cc:330] T e1b656691604454791305c19f0cbf844 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: stopping tablet replica
I20260620 20:22:17.722141 2597 raft_consensus.cc:2229] T e1b656691604454791305c19f0cbf844 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:22:17.723057 2597 raft_consensus.cc:2258] T e1b656691604454791305c19f0cbf844 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:17.723804 2599 tablet_replica.cc:330] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687: stopping tablet replica
I20260620 20:22:17.724040 2598 tablet_replica.cc:330] T e1b656691604454791305c19f0cbf844 P 5df6f872d433441d909995caf387b602: stopping tablet replica
I20260620 20:22:17.724066 2599 raft_consensus.cc:2229] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:22:17.724267 2598 raft_consensus.cc:2229] T e1b656691604454791305c19f0cbf844 P 5df6f872d433441d909995caf387b602 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:22:17.724546 2598 raft_consensus.cc:2258] T e1b656691604454791305c19f0cbf844 P 5df6f872d433441d909995caf387b602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:17.724555 2597 ts_tablet_manager.cc:1905] T e1b656691604454791305c19f0cbf844 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:17.724965 2599 raft_consensus.cc:2258] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:17.725584 2598 ts_tablet_manager.cc:1905] T e1b656691604454791305c19f0cbf844 P 5df6f872d433441d909995caf387b602: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:17.726387 2599 ts_tablet_manager.cc:1905] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:17.731432 2597 ts_tablet_manager.cc:1918] T e1b656691604454791305c19f0cbf844 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:22:17.731588 2597 log.cc:1196] T e1b656691604454791305c19f0cbf844 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/e1b656691604454791305c19f0cbf844
I20260620 20:22:17.732057 2597 ts_tablet_manager.cc:1939] T e1b656691604454791305c19f0cbf844 P 6bc4886fe4cd4b06aa4be3f7e4c1c31a: Deleting consensus metadata
I20260620 20:22:17.732228 2598 ts_tablet_manager.cc:1918] T e1b656691604454791305c19f0cbf844 P 5df6f872d433441d909995caf387b602: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:22:17.732379 2598 log.cc:1196] T e1b656691604454791305c19f0cbf844 P 5df6f872d433441d909995caf387b602: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/e1b656691604454791305c19f0cbf844
I20260620 20:22:17.732841 2598 ts_tablet_manager.cc:1939] T e1b656691604454791305c19f0cbf844 P 5df6f872d433441d909995caf387b602: Deleting consensus metadata
I20260620 20:22:17.733208 1941 catalog_manager.cc:4841] TS 6bc4886fe4cd4b06aa4be3f7e4c1c31a (127.31.235.129:36535): tablet e1b656691604454791305c19f0cbf844 (table test-table [id=f379f944108343ebad2dbb375a2a2712]) successfully deleted
I20260620 20:22:17.733265 2599 ts_tablet_manager.cc:1918] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 20:22:17.733515 2599 log.cc:1196] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/e1b656691604454791305c19f0cbf844
I20260620 20:22:17.734019 2599 ts_tablet_manager.cc:1939] T e1b656691604454791305c19f0cbf844 P 777d22cfe7b942d0b1b0f8a994ebf687: Deleting consensus metadata
I20260620 20:22:17.734023 1941 catalog_manager.cc:4841] TS 5df6f872d433441d909995caf387b602 (127.31.235.131:46523): tablet e1b656691604454791305c19f0cbf844 (table test-table [id=f379f944108343ebad2dbb375a2a2712]) successfully deleted
Jun 20 20:22:17 dist-test-slave-zzwf krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986935, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:22:17.734995 1941 catalog_manager.cc:4841] TS 777d22cfe7b942d0b1b0f8a994ebf687 (127.31.235.130:34217): tablet e1b656691604454791305c19f0cbf844 (table test-table [id=f379f944108343ebad2dbb375a2a2712]) successfully deleted
I20260620 20:22:17.741860 2601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:17.712388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57638 (local address 127.31.235.129:36535)
0620 20:22:17.717850 (+ 5462us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:17.717859 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:17.724000 (+ 6141us) server_negotiation.cc:407] Connection header received
0620 20:22:17.724533 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:17.724546 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:17.724638 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:17.724825 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:17.726735 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:17.727872 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:17.729772 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:17.730480 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:17.730579 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:17.735977 (+ 5398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:17.735993 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:17.736010 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:17.736045 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:17.738861 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:17.739606 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:17.739613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:17.739615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:17.739741 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:17.740241 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:17.740247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:17.740250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:17.740611 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:17.740915 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:17.741229 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 20:22:17.741460 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5301,"thread_start_us":5158,"threads_started":1}
W20260620 20:22:17.742482 2157 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57638
W20260620 20:22:17.743089 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 2026: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:22:17.743335 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 2026
I20260620 20:22:17.767213 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 2166
Jun 20 20:22:17 dist-test-slave-zzwf krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986935, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.235.130@KRBTEST.COM
I20260620 20:22:17.782997 2602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:17.768076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43996 (local address 127.31.235.130:34217)
0620 20:22:17.768642 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:17.768648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:17.768666 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:22:17.768729 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:17.768734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:17.768809 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:17.768931 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:17.770240 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:17.771084 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:17.772322 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:17.772723 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:17.772818 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:17.777056 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:17.777069 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:17.777085 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:17.777127 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:17.780289 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:17.780969 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:17.780976 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:17.780979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:17.781110 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:17.781467 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:17.781470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:17.781472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:17.781906 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:17.782130 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:17.782472 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 20:22:17.782636 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":250,"threads_started":1}
W20260620 20:22:17.783707 2296 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43996
W20260620 20:22:17.784389 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 2166: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:22:17.784528 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 2166
I20260620 20:22:17.809562 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 2305
Jun 20 20:22:17 dist-test-slave-zzwf krb5kdc[1910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986935, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.235.131@KRBTEST.COM
I20260620 20:22:17.824628 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:17.810326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52928 (local address 127.31.235.131:46523)
0620 20:22:17.810797 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:17.810801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:17.810818 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:22:17.810871 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:17.810875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:17.810938 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:17.811030 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:17.812295 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:17.813076 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:17.814380 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:17.814775 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:17.814949 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:17.819403 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:17.819420 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:17.819437 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:17.819473 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:17.822341 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:17.823015 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:17.823021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:17.823023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:17.823145 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:17.823450 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:17.823453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:17.823455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:17.823766 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:17.823978 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:17.824193 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 20:22:17.824362 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":228,"threads_started":1}
W20260620 20:22:17.825273 2435 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52928
W20260620 20:22:17.825973 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 2305: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:22:17.826144 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 2305
I20260620 20:22:17.843344 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 1923
I20260620 20:22:17.844589 1995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:17.962663 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 1923
2026-06-20T20:22:17Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8386 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:18 dist-test-slave-zzwf krb5kdc[2607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:18 dist-test-slave-zzwf krb5kdc[2607](info): set up 2 sockets
Jun 20 20:22:18 dist-test-slave-zzwf krb5kdc[2607](info): commencing operation
krb5kdc: starting...
W20260620 20:22:19.142004 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 20:22:19.357416 32686 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.215s user 0.000s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 20:22:19.617095 32686 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.259s user 0.002s sys 0.003s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 20:22:19.916658 32686 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.299s user 0.004s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/test-user.keytab.
W20260620 20:22:20.732822 32686 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.816s user 0.000s sys 0.004s
Jun 20 20:22:20 dist-test-slave-zzwf krb5kdc[2607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
W20260620 20:22:22.089074 32686 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.31.235.190: real 1.305s user 0.000s sys 0.008s
I20260620 20:22:22.090183 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:37419
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:41263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:37419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:22.449361 2620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:22.449797 2620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:22.449909 2620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:22.457893 2620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:22.458021 2620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:22.458069 2620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:22.458119 2620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:22.458164 2620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:22.468469 2620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:41263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:37419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:37419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:22.469844 2620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:22.471180 2620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:22.484508 2625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:22.484788 2626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:22.484860 2628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:22.485809 2620 server_base.cc:1034] running on GCE node
I20260620 20:22:22.486763 2620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:22.488464 2620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:22.489696 2620 hybrid_clock.cc:648] HybridClock initialized: now 1781986942489656 us; error 55 us; skew 500 ppm
I20260620 20:22:22.490073 2620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:22 dist-test-slave-zzwf krb5kdc[2607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986942, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:22.493371 2620 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:22:22.494895 2620 webserver.cc:460] Webserver started at http://127.31.235.190:39839/ using document root <none> and password file <none>
I20260620 20:22:22.495646 2620 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:22.495787 2620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:22.496098 2620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:22.499325 2620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "8c40f628813844e0bf6f3bda85cccd96"
format_stamp: "Formatted at 2026-06-20 20:22:22 on dist-test-slave-zzwf"
server_key: "71c65d72027f90cdf2a9497b7af9c60a"
server_key_iv: "a66d6459b668d4b24773e451482964a9"
server_key_version: "encryptionkey@0"
I20260620 20:22:22.500088 2620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "8c40f628813844e0bf6f3bda85cccd96"
format_stamp: "Formatted at 2026-06-20 20:22:22 on dist-test-slave-zzwf"
server_key: "71c65d72027f90cdf2a9497b7af9c60a"
server_key_iv: "a66d6459b668d4b24773e451482964a9"
server_key_version: "encryptionkey@0"
I20260620 20:22:22.505638 2620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 20:22:22.509351 2635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:22.510946 2620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:22:22.511153 2620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "8c40f628813844e0bf6f3bda85cccd96"
format_stamp: "Formatted at 2026-06-20 20:22:22 on dist-test-slave-zzwf"
server_key: "71c65d72027f90cdf2a9497b7af9c60a"
server_key_iv: "a66d6459b668d4b24773e451482964a9"
server_key_version: "encryptionkey@0"
I20260620 20:22:22.511364 2620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:22.538439 2620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:22.542691 2620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:22.542990 2620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:22.561757 2620 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:37419
I20260620 20:22:22.561764 2686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:37419 every 8 connection(s)
I20260620 20:22:22.563180 2620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:22.567144 2687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:22.569769 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 2620
I20260620 20:22:22.569952 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:22:22.570605 32686 external_mini_cluster.cc:1428] Setting key 5bec77582855bae7d883635150d3ec20
I20260620 20:22:22.578794 2687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96: Bootstrap starting.
I20260620 20:22:22.582495 2687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:22.583640 2687 log.cc:824] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:22:22 dist-test-slave-zzwf krb5kdc[2607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:22.586882 2687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96: No bootstrap required, opened a new log
I20260620 20:22:22.595976 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:22.572617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37248 (local address 127.31.235.190:37419)
0620 20:22:22.573337 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:22.573350 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:22.573389 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:22:22.575151 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:22.575183 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:22.576296 (+ 1113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:22.576760 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:22.578351 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:22.579743 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:22.581332 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:22.582425 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:22.582632 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:22.587112 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:22.587123 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:22.587143 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:22.587200 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:22.591333 (+ 4133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:22.592038 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:22.592048 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:22.592057 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:22.592254 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:22.592571 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:22.592576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:22.592579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:22.593340 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:22.593692 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:22.594048 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 20:22:22.594530 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":319,"threads_started":1}
I20260620 20:22:22.598030 2687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c40f628813844e0bf6f3bda85cccd96" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 37419 } }
I20260620 20:22:22.598484 2687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:22.598618 2687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c40f628813844e0bf6f3bda85cccd96, State: Initialized, Role: FOLLOWER
I20260620 20:22:22.599279 2687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c40f628813844e0bf6f3bda85cccd96" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 37419 } }
I20260620 20:22:22.599524 2687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:22.599655 2687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:22.599820 2687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:22.606180 2687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c40f628813844e0bf6f3bda85cccd96" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 37419 } }
I20260620 20:22:22.606839 2687 leader_election.cc:302] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c40f628813844e0bf6f3bda85cccd96; no voters:
I20260620 20:22:22.608152 2687 leader_election.cc:288] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:22.608296 2692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:22.610325 2687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:22.610457 2692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96 [term 1 LEADER]: Becoming Leader. State: Replica: 8c40f628813844e0bf6f3bda85cccd96, State: Running, Role: LEADER
I20260620 20:22:22.611032 2692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c40f628813844e0bf6f3bda85cccd96" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 37419 } }
I20260620 20:22:22.614063 2694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c40f628813844e0bf6f3bda85cccd96. Latest consensus state: current_term: 1 leader_uuid: "8c40f628813844e0bf6f3bda85cccd96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c40f628813844e0bf6f3bda85cccd96" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 37419 } } }
I20260620 20:22:22.614230 2693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c40f628813844e0bf6f3bda85cccd96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c40f628813844e0bf6f3bda85cccd96" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 37419 } } }
I20260620 20:22:22.614497 2694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:22.614524 2693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:22.616195 2701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:22.620733 2701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:22.628739 2701 catalog_manager.cc:1347] Generated new cluster ID: c80981c58a8e488981d572f4bc25f5ba
I20260620 20:22:22.628850 2701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:22.655297 2701 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:22.656656 2701 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:22.664335 2701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c40f628813844e0bf6f3bda85cccd96: Generated new TSK 0
I20260620 20:22:22.665293 2701 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
W20260620 20:22:23.671131 32686 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.31.235.129: real 0.990s user 0.002s sys 0.009s
I20260620 20:22:23.672117 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:37419
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:41263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:24.002203 2713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:24.002542 2713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:24.002732 2713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:24.011197 2713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:24.011344 2713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:24.011540 2713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:22:24.022580 2713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:41263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:37419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:24.023919 2713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:24.025219 2713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:24.038807 2718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:24.038924 2719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:24.038934 2721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:24.039963 2713 server_base.cc:1034] running on GCE node
I20260620 20:22:24.040601 2713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:24.041664 2713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:24.042857 2713 hybrid_clock.cc:648] HybridClock initialized: now 1781986944042788 us; error 61 us; skew 500 ppm
I20260620 20:22:24.043169 2713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:24 dist-test-slave-zzwf krb5kdc[2607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986944, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:24.050177 2713 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:22:24.051683 2713 webserver.cc:460] Webserver started at http://127.31.235.129:37443/ using document root <none> and password file <none>
I20260620 20:22:24.052380 2713 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:24.052481 2713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:24.052812 2713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:24.055724 2713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "3ef7ad4fec444d93a372f62b1b3e4920"
format_stamp: "Formatted at 2026-06-20 20:22:24 on dist-test-slave-zzwf"
server_key: "b65b260bd8afbe4c691bbad86804bf51"
server_key_iv: "796fb61699f7c911ece2212e59cafc03"
server_key_version: "encryptionkey@0"
I20260620 20:22:24.056484 2713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "3ef7ad4fec444d93a372f62b1b3e4920"
format_stamp: "Formatted at 2026-06-20 20:22:24 on dist-test-slave-zzwf"
server_key: "b65b260bd8afbe4c691bbad86804bf51"
server_key_iv: "796fb61699f7c911ece2212e59cafc03"
server_key_version: "encryptionkey@0"
I20260620 20:22:24.061448 2713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 20:22:24.064949 2728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:24.066253 2713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:24.066419 2713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "3ef7ad4fec444d93a372f62b1b3e4920"
format_stamp: "Formatted at 2026-06-20 20:22:24 on dist-test-slave-zzwf"
server_key: "b65b260bd8afbe4c691bbad86804bf51"
server_key_iv: "796fb61699f7c911ece2212e59cafc03"
server_key_version: "encryptionkey@0"
I20260620 20:22:24.066615 2713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:24.081712 2713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:24.086062 2713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:24.086366 2713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:24.087329 2713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:24.088663 2713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:24.088783 2713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:24.088923 2713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:24.088997 2713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:24.122334 2713 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:33249
I20260620 20:22:24.122365 2840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:33249 every 8 connection(s)
I20260620 20:22:24.123611 2713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:22:24.131353 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 2713
I20260620 20:22:24.131561 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:22:24.132302 32686 external_mini_cluster.cc:1428] Setting key 9c710c21f2859466433190f2422e957b
Jun 20 20:22:24 dist-test-slave-zzwf krb5kdc[2607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986944, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:24.144335 2843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:24.126222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:55739 (local address 127.31.235.190:37419)
0620 20:22:24.126638 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:24.126642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:24.127625 (+ 983us) server_negotiation.cc:407] Connection header received
0620 20:22:24.129404 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:24.129408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:24.129480 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:24.129624 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:24.131027 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:24.131828 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:24.133898 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:24.134547 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:24.134677 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:24.138844 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:24.138849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:24.138852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:24.138883 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:24.141405 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:24.142122 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:24.142129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:24.142133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:24.142255 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:24.142661 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:24.142666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:24.142668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:24.142931 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:24.143088 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:24.143752 (+ 664us) server_negotiation.cc:299] Negotiation successful
0620 20:22:24.143962 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":216,"threads_started":1}
I20260620 20:22:24.145287 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:24.126794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:37419 (local address 127.31.235.129:55739)
0620 20:22:24.127467 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:24.127507 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:24.129149 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:24.129833 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:24.129841 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:24.130328 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:24.130860 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:24.130875 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:24.131974 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:24.131979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:24.133763 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:24.133774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:24.134775 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:24.134783 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:24.135069 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:24.135814 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:24.135850 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:24.138608 (+ 2758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:24.141556 (+ 2948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:24.141567 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:24.141587 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:24.142004 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:24.142358 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:24.142364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:24.142367 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:24.142546 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:24.143052 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:24.143060 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:24.143407 (+ 347us) client_negotiation.cc:769] Sending connection context
0620 20:22:24.143654 (+ 247us) client_negotiation.cc:240] Negotiation successful
0620 20:22:24.144167 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":269,"threads_started":1}
I20260620 20:22:24.146924 2841 heartbeater.cc:347] Connected to a master server at 127.31.235.190:37419
I20260620 20:22:24.147274 2841 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:24.148209 2841 heartbeater.cc:511] Master 127.31.235.190:37419 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.235.130@KRBTEST.COM; defaulting to no policy
I20260620 20:22:24.150884 2652 ts_manager.cc:194] Registered new tserver with Master: 3ef7ad4fec444d93a372f62b1b3e4920 (127.31.235.129:33249)
I20260620 20:22:24.152410 2652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:55739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
I20260620 20:22:24.177860 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:37419
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:41263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:24.504473 2847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:24.504774 2847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:24.504873 2847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:24.512374 2847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:24.512492 2847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:24.512605 2847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:22:24.522728 2847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:41263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:37419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:24.524044 2847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:24.525295 2847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:24.538455 2852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:24.538403 2855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:24.538371 2853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:24.539417 2847 server_base.cc:1034] running on GCE node
I20260620 20:22:24.540019 2847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:24.540954 2847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:24.542169 2847 hybrid_clock.cc:648] HybridClock initialized: now 1781986944542117 us; error 44 us; skew 500 ppm
I20260620 20:22:24.542485 2847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:24 dist-test-slave-zzwf krb5kdc[2607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986944, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:24.545681 2847 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:22:24.547024 2847 webserver.cc:460] Webserver started at http://127.31.235.130:35843/ using document root <none> and password file <none>
I20260620 20:22:24.547679 2847 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:24.547773 2847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:24.548017 2847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:24.550922 2847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "f00943c1780247dfb00cf41d31a16ba3"
format_stamp: "Formatted at 2026-06-20 20:22:24 on dist-test-slave-zzwf"
server_key: "0cbec7cb2b72ab55f19735d108eba003"
server_key_iv: "543c3420a749beedf24ad5ac55350b76"
server_key_version: "encryptionkey@0"
I20260620 20:22:24.551627 2847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "f00943c1780247dfb00cf41d31a16ba3"
format_stamp: "Formatted at 2026-06-20 20:22:24 on dist-test-slave-zzwf"
server_key: "0cbec7cb2b72ab55f19735d108eba003"
server_key_iv: "543c3420a749beedf24ad5ac55350b76"
server_key_version: "encryptionkey@0"
I20260620 20:22:24.556588 2847 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:22:24.559950 2862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:24.561305 2847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:22:24.561589 2847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "f00943c1780247dfb00cf41d31a16ba3"
format_stamp: "Formatted at 2026-06-20 20:22:24 on dist-test-slave-zzwf"
server_key: "0cbec7cb2b72ab55f19735d108eba003"
server_key_iv: "543c3420a749beedf24ad5ac55350b76"
server_key_version: "encryptionkey@0"
I20260620 20:22:24.561790 2847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:24.592485 2847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:24.597098 2847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:24.597391 2847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:24.598361 2847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:24.599632 2847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:24.599731 2847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:24.599875 2847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:24.599944 2847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:24.632373 2847 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:43281
I20260620 20:22:24.632421 2974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:43281 every 8 connection(s)
I20260620 20:22:24.633747 2847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:22:24.635826 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 2847
I20260620 20:22:24.636018 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:22:24.636597 32686 external_mini_cluster.cc:1428] Setting key 2694ede10158817fdbbd1ffb22c18a29
Jun 20 20:22:24 dist-test-slave-zzwf krb5kdc[2607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986944, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:24.653816 2843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:24.636289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:57671 (local address 127.31.235.190:37419)
0620 20:22:24.636479 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:24.636489 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:24.637953 (+ 1464us) server_negotiation.cc:407] Connection header received
0620 20:22:24.639748 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:24.639753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:24.639854 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:24.639965 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:24.641413 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:24.642173 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:24.643670 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:24.644115 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:24.644245 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:24.647877 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:24.647882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:24.647886 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:24.647915 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:24.650429 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:24.651412 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:24.651416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:24.651418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:24.651537 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:24.652051 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:24.652055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:24.652057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:24.652305 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:24.652478 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:24.653317 (+ 839us) server_negotiation.cc:299] Negotiation successful
0620 20:22:24.653566 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.31.235.131@KRBTEST.COM; defaulting to no policy
I20260620 20:22:24.655232 2977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:24.636940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:37419 (local address 127.31.235.130:57671)
0620 20:22:24.637803 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:24.637847 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:24.639494 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:24.640183 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:24.640191 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:24.640723 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:24.641247 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:24.641262 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:24.642340 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:24.642344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:24.643527 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:24.643538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:24.644330 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:24.644347 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:24.644562 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:24.645214 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:24.645239 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:24.647663 (+ 2424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:24.650581 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:24.650593 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:24.650614 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:24.651276 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:24.651654 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:24.651661 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:24.651667 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:24.651942 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:24.652459 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:24.652466 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:24.652935 (+ 469us) client_negotiation.cc:769] Sending connection context
0620 20:22:24.653277 (+ 342us) client_negotiation.cc:240] Negotiation successful
0620 20:22:24.653888 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":376,"threads_started":1}
I20260620 20:22:24.656555 2975 heartbeater.cc:347] Connected to a master server at 127.31.235.190:37419
I20260620 20:22:24.656857 2975 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.131@KRBTEST.COM" created.
I20260620 20:22:24.657792 2975 heartbeater.cc:511] Master 127.31.235.190:37419 requested a full tablet report, sending...
I20260620 20:22:24.659222 2652 ts_manager.cc:194] Registered new tserver with Master: f00943c1780247dfb00cf41d31a16ba3 (127.31.235.130:43281)
I20260620 20:22:24.660018 2652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:57671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
I20260620 20:22:24.680902 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:37419
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:41263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:25.009208 2980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:25.009598 2980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:25.009759 2980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:25.017400 2980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:25.017584 2980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:25.017755 2980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:22:25.027486 2980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:41263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:37419
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:25.028743 2980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:25.030092 2980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:25.043283 2986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:25.043360 2985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:25.043325 2988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:25.044248 2980 server_base.cc:1034] running on GCE node
I20260620 20:22:25.044865 2980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:25.045976 2980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:25.047191 2980 hybrid_clock.cc:648] HybridClock initialized: now 1781986945047128 us; error 61 us; skew 500 ppm
I20260620 20:22:25.047504 2980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:25 dist-test-slave-zzwf krb5kdc[2607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986945, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:25.050629 2980 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:22:25.052032 2980 webserver.cc:460] Webserver started at http://127.31.235.131:33631/ using document root <none> and password file <none>
I20260620 20:22:25.052736 2980 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:25.052902 2980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:25.053197 2980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:25.056106 2980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "7a486fa1af8547e3bd1b0c2f7fd4e83c"
format_stamp: "Formatted at 2026-06-20 20:22:25 on dist-test-slave-zzwf"
server_key: "07855a34d1cd15825704bae391da57b2"
server_key_iv: "49a9c9f93f0f0a41414df065e8e38259"
server_key_version: "encryptionkey@0"
I20260620 20:22:25.056897 2980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "7a486fa1af8547e3bd1b0c2f7fd4e83c"
format_stamp: "Formatted at 2026-06-20 20:22:25 on dist-test-slave-zzwf"
server_key: "07855a34d1cd15825704bae391da57b2"
server_key_iv: "49a9c9f93f0f0a41414df065e8e38259"
server_key_version: "encryptionkey@0"
I20260620 20:22:25.063059 2980 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.008s
I20260620 20:22:25.066839 2995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:25.068414 2980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:22:25.068596 2980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "7a486fa1af8547e3bd1b0c2f7fd4e83c"
format_stamp: "Formatted at 2026-06-20 20:22:25 on dist-test-slave-zzwf"
server_key: "07855a34d1cd15825704bae391da57b2"
server_key_iv: "49a9c9f93f0f0a41414df065e8e38259"
server_key_version: "encryptionkey@0"
I20260620 20:22:25.068830 2980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:25.130626 2980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:25.134747 2980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:25.135044 2980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:25.135977 2980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:25.137274 2980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:25.137373 2980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:25.137545 2980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:25.137627 2980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260620 20:22:25.155448 2841 heartbeater.cc:503] Master 127.31.235.190:37419 was elected leader, sending a full tablet report...
I20260620 20:22:25.172952 2980 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:36309
I20260620 20:22:25.173013 3107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:36309 every 8 connection(s)
I20260620 20:22:25.174551 2980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:22:25.180357 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 2980
I20260620 20:22:25.180564 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:22:25.181603 32686 external_mini_cluster.cc:1428] Setting key 2daf701efbe73fa87d2e90c9bbf07d98
Jun 20 20:22:25 dist-test-slave-zzwf krb5kdc[2607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986945, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:25.198133 3110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:25.177579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:42495 (local address 127.31.235.190:37419)
0620 20:22:25.178017 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:25.178022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:25.179116 (+ 1094us) server_negotiation.cc:407] Connection header received
0620 20:22:25.181195 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:25.181205 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:25.181343 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:25.181762 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:25.183577 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:25.184908 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:25.186955 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:25.187499 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:25.187684 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:25.191448 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:25.191454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:25.191458 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:25.191497 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:25.194338 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:25.195210 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:25.195215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:25.195217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:25.195358 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:25.195794 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:25.195798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:25.195800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:25.196179 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:25.196378 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:25.197481 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0620 20:22:25.197735 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":219,"threads_started":1}
I20260620 20:22:25.199162 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:25.178169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:37419 (local address 127.31.235.131:42495)
0620 20:22:25.178938 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:25.178985 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:25.180730 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:25.182011 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:25.182019 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:25.182617 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:25.183337 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:25.183357 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:25.185112 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:25.185147 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:25.186722 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:25.186735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:25.187779 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:25.187785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:25.188032 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:25.188754 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:25.188778 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:25.191184 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:25.194499 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:25.194506 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:25.194520 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:25.195022 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:25.195463 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:25.195466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:25.195469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:25.195656 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:25.196505 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:25.196519 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:25.197084 (+ 565us) client_negotiation.cc:769] Sending connection context
0620 20:22:25.197362 (+ 278us) client_negotiation.cc:240] Negotiation successful
0620 20:22:25.197971 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":294,"threads_started":1}
I20260620 20:22:25.200627 3108 heartbeater.cc:347] Connected to a master server at 127.31.235.190:37419
I20260620 20:22:25.200980 3108 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:25.201792 3108 heartbeater.cc:511] Master 127.31.235.190:37419 requested a full tablet report, sending...
I20260620 20:22:25.203627 2652 ts_manager.cc:194] Registered new tserver with Master: 7a486fa1af8547e3bd1b0c2f7fd4e83c (127.31.235.131:36309)
I20260620 20:22:25.204686 2652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:42495
I20260620 20:22:25.211489 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 20:22:25.243178 3110 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:25.240535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.31.235.190:37419)
0620 20:22:25.240784 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:25.240793 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:25.241144 (+ 351us) server_negotiation.cc:407] Connection header received
0620 20:22:25.241314 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:25.241318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:25.241418 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:25.241574 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:25.242912 (+ 1338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37254: BlockingRecv error: recv got EOF from 127.0.0.1:37254 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:22:25.246456 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 2713
W20260620 20:22:25.249627 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 2713: Not authorized: Client connection negotiation failed: client connection to 127.31.235.129:33249: server requires authentication, but client does not have Kerberos credentials available
I20260620 20:22:25.249815 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 2713
I20260620 20:22:25.265096 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 2847
W20260620 20:22:25.267719 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 2847: Not authorized: Client connection negotiation failed: client connection to 127.31.235.130:43281: server requires authentication, but client does not have Kerberos credentials available
I20260620 20:22:25.267900 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 2847
I20260620 20:22:25.283468 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 2980
W20260620 20:22:25.286650 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 2980: Not authorized: Client connection negotiation failed: client connection to 127.31.235.131:36309: server requires authentication, but client does not have Kerberos credentials available
I20260620 20:22:25.286772 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 2980
I20260620 20:22:25.302040 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 2620
I20260620 20:22:25.303234 2682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:25.406394 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 2620
2026-06-20T20:22:25Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (7439 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:25 dist-test-slave-zzwf krb5kdc[3126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:25 dist-test-slave-zzwf krb5kdc[3126](info): set up 2 sockets
Jun 20 20:22:25 dist-test-slave-zzwf krb5kdc[3126](info): commencing operation
krb5kdc: starting...
W20260620 20:22:26.494202 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:22:26 dist-test-slave-zzwf krb5kdc[3126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:22:26.689656 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:39863
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:39863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:27.019517 3139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:27.019850 3139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:27.019991 3139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:27.027462 3139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:27.027585 3139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:27.027631 3139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:27.027680 3139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:27.027727 3139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:27.037796 3139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:39863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:39863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:27.039108 3139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:27.040364 3139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:27.052601 3147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:27.052588 3144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:27.052577 3145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:27.053193 3139 server_base.cc:1034] running on GCE node
I20260620 20:22:27.054102 3139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:27.055593 3139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:27.056828 3139 hybrid_clock.cc:648] HybridClock initialized: now 1781986947056765 us; error 63 us; skew 500 ppm
I20260620 20:22:27.057219 3139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:27 dist-test-slave-zzwf krb5kdc[3126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986947, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:27.060490 3139 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:22:27.062196 3139 webserver.cc:460] Webserver started at http://127.31.235.190:45275/ using document root <none> and password file <none>
I20260620 20:22:27.062865 3139 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:27.062983 3139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:27.063235 3139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:27.066298 3139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "cb00b078afa440be972ed4690d5daaf3"
format_stamp: "Formatted at 2026-06-20 20:22:27 on dist-test-slave-zzwf"
server_key: "8188e7ea72c7a5a0c32a5dab6f47e608"
server_key_iv: "f5f7201de8bbbb274826fa0e7b4c2bb2"
server_key_version: "encryptionkey@0"
I20260620 20:22:27.067135 3139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "cb00b078afa440be972ed4690d5daaf3"
format_stamp: "Formatted at 2026-06-20 20:22:27 on dist-test-slave-zzwf"
server_key: "8188e7ea72c7a5a0c32a5dab6f47e608"
server_key_iv: "f5f7201de8bbbb274826fa0e7b4c2bb2"
server_key_version: "encryptionkey@0"
I20260620 20:22:27.072993 3139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:22:27.076761 3154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:27.078368 3139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:27.078553 3139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "cb00b078afa440be972ed4690d5daaf3"
format_stamp: "Formatted at 2026-06-20 20:22:27 on dist-test-slave-zzwf"
server_key: "8188e7ea72c7a5a0c32a5dab6f47e608"
server_key_iv: "f5f7201de8bbbb274826fa0e7b4c2bb2"
server_key_version: "encryptionkey@0"
I20260620 20:22:27.078759 3139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:27.098328 3139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:27.102658 3139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:27.102952 3139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:27.121173 3139 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:39863
I20260620 20:22:27.121207 3205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:39863 every 8 connection(s)
I20260620 20:22:27.122561 3139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:27.126718 3206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:27.128343 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 3139
I20260620 20:22:27.128572 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:22:27.129199 32686 external_mini_cluster.cc:1428] Setting key aba2cdc058ed8f8ae9007781456dcc22
I20260620 20:22:27.142257 3206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3: Bootstrap starting.
Jun 20 20:22:27 dist-test-slave-zzwf krb5kdc[3126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:27.146329 3206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:27.147919 3206 log.cc:824] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:27.151749 3206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3: No bootstrap required, opened a new log
I20260620 20:22:27.153683 3209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:27.131057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39600 (local address 127.31.235.190:39863)
0620 20:22:27.131876 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:27.131890 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:27.131936 (+ 46us) server_negotiation.cc:407] Connection header received
0620 20:22:27.133588 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:27.133612 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:27.134055 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:27.134394 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:27.136319 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:27.137680 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:27.139680 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:27.140362 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:27.140494 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:27.145305 (+ 4811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:27.145318 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:27.145341 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:27.145391 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:27.149418 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:27.150181 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:27.150189 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:27.150198 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:27.150404 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:27.150755 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:27.150761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:27.150766 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:27.151490 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:27.151773 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:27.152188 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 20:22:27.152618 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":396,"threads_started":1}
I20260620 20:22:27.157315 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:27.159691 3206 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb00b078afa440be972ed4690d5daaf3" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39863 } }
I20260620 20:22:27.160056 3206 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:27.160164 3206 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb00b078afa440be972ed4690d5daaf3, State: Initialized, Role: FOLLOWER
I20260620 20:22:27.160730 3206 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb00b078afa440be972ed4690d5daaf3" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39863 } }
I20260620 20:22:27.160941 3206 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:27.161067 3206 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:27.161221 3206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:27.163739 3206 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb00b078afa440be972ed4690d5daaf3" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39863 } }
I20260620 20:22:27.164345 3206 leader_election.cc:302] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb00b078afa440be972ed4690d5daaf3; no voters:
I20260620 20:22:27.164985 3206 leader_election.cc:288] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:27.165112 3212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:27.166366 3212 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3 [term 1 LEADER]: Becoming Leader. State: Replica: cb00b078afa440be972ed4690d5daaf3, State: Running, Role: LEADER
I20260620 20:22:27.166935 3212 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb00b078afa440be972ed4690d5daaf3" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39863 } }
I20260620 20:22:27.167264 3206 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:27.169570 3214 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb00b078afa440be972ed4690d5daaf3. Latest consensus state: current_term: 1 leader_uuid: "cb00b078afa440be972ed4690d5daaf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb00b078afa440be972ed4690d5daaf3" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39863 } } }
I20260620 20:22:27.169965 3214 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:27.170555 3213 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb00b078afa440be972ed4690d5daaf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb00b078afa440be972ed4690d5daaf3" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39863 } } }
I20260620 20:22:27.170825 3213 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:27.172582 3219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:27.184224 3219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:27.198881 3219 catalog_manager.cc:1347] Generated new cluster ID: db9aee57c3814a7d9d218df961478971
I20260620 20:22:27.199285 3219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:27.233667 3209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:27.224689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39602 (local address 127.31.235.190:39863)
0620 20:22:27.224884 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:27.224892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:27.225318 (+ 426us) server_negotiation.cc:407] Connection header received
0620 20:22:27.225531 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:27.225537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:27.225616 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:27.225716 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:27.227013 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:27.228096 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:27.229985 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:27.230658 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:27.230789 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:27.232024 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:27.232030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:27.232035 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:27.232836 (+ 801us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:27.232907 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:27.232942 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:27.233042 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:27.233204 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 20:22:27.233292 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 20:22:27.246815 3219 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:27.248577 3219 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:27.257424 3219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb00b078afa440be972ed4690d5daaf3: Generated new TSK 0
I20260620 20:22:27.258347 3219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:27.273020 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 3139
I20260620 20:22:27.274339 3201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:27.367187 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 3139
2026-06-20T20:22:27Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1956 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:27 dist-test-slave-zzwf krb5kdc[3240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:27 dist-test-slave-zzwf krb5kdc[3240](info): set up 2 sockets
Jun 20 20:22:27 dist-test-slave-zzwf krb5kdc[3240](info): commencing operation
krb5kdc: starting...
W20260620 20:22:28.442495 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:22:28 dist-test-slave-zzwf krb5kdc[3240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:22:28.633399 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:36243
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:35717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:36243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:28.949569 3253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:28.949867 3253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:28.949965 3253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:28.957090 3253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:28.957211 3253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:28.957257 3253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:28.957306 3253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:28.957353 3253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:28.967036 3253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:35717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:36243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:36243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:28.968312 3253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:28.969480 3253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:28.981470 3258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:28.981527 3259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:28.981469 3261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:28.987174 3253 server_base.cc:1034] running on GCE node
I20260620 20:22:28.988065 3253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:28.989688 3253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:28.990924 3253 hybrid_clock.cc:648] HybridClock initialized: now 1781986948990888 us; error 49 us; skew 500 ppm
I20260620 20:22:28.991302 3253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:28 dist-test-slave-zzwf krb5kdc[3240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986948, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:28.994536 3253 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:22:28.996182 3253 webserver.cc:460] Webserver started at http://127.31.235.190:35729/ using document root <none> and password file <none>
I20260620 20:22:28.996903 3253 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:28.997051 3253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:28.997354 3253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:29.000478 3253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "8c50affd32784678a59d44bfffe22515"
format_stamp: "Formatted at 2026-06-20 20:22:28 on dist-test-slave-zzwf"
server_key: "1c154bdce9eefa9ccb3232036a50a708"
server_key_iv: "9c212b8a13a6f269757acab5ae1a55f8"
server_key_version: "encryptionkey@0"
I20260620 20:22:29.001262 3253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "8c50affd32784678a59d44bfffe22515"
format_stamp: "Formatted at 2026-06-20 20:22:28 on dist-test-slave-zzwf"
server_key: "1c154bdce9eefa9ccb3232036a50a708"
server_key_iv: "9c212b8a13a6f269757acab5ae1a55f8"
server_key_version: "encryptionkey@0"
I20260620 20:22:29.006736 3253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 20:22:29.010421 3268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:29.011799 3253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:22:29.012001 3253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "8c50affd32784678a59d44bfffe22515"
format_stamp: "Formatted at 2026-06-20 20:22:28 on dist-test-slave-zzwf"
server_key: "1c154bdce9eefa9ccb3232036a50a708"
server_key_iv: "9c212b8a13a6f269757acab5ae1a55f8"
server_key_version: "encryptionkey@0"
I20260620 20:22:29.012203 3253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:29.023336 3253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:29.027513 3253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:29.027824 3253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:29.045941 3253 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:36243
I20260620 20:22:29.045969 3319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:36243 every 8 connection(s)
I20260620 20:22:29.047340 3253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:29.051105 3320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:29.051347 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 3253
I20260620 20:22:29.051540 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:22:29.052171 32686 external_mini_cluster.cc:1428] Setting key 363f61f6c3c4d0b6e1181829407a8d22
I20260620 20:22:29.064730 3320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515: Bootstrap starting.
I20260620 20:22:29.067811 3320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:29.068830 3320 log.cc:824] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:22:29 dist-test-slave-zzwf krb5kdc[3240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:29.072433 3320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515: No bootstrap required, opened a new log
I20260620 20:22:29.081106 3323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.054056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44446 (local address 127.31.235.190:36243)
0620 20:22:29.054828 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.054841 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.054881 (+ 40us) server_negotiation.cc:407] Connection header received
0620 20:22:29.056910 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.056934 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.057346 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.057810 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.059492 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.060741 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.062609 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.063379 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.063522 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:29.073052 (+ 9530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:29.073061 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:29.073077 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:29.073120 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:29.076829 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.077638 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.077648 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.077662 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.077848 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.078240 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.078253 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.078263 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.078834 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:29.079180 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:29.079568 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 20:22:29.080075 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":365,"threads_started":1}
I20260620 20:22:29.080838 3320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c50affd32784678a59d44bfffe22515" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36243 } }
I20260620 20:22:29.081301 3320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:29.081444 3320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c50affd32784678a59d44bfffe22515, State: Initialized, Role: FOLLOWER
I20260620 20:22:29.082059 3320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c50affd32784678a59d44bfffe22515" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36243 } }
I20260620 20:22:29.082320 3320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:29.082469 3320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:29.082665 3320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:29.084923 3320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c50affd32784678a59d44bfffe22515" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36243 } }
I20260620 20:22:29.085427 3320 leader_election.cc:302] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c50affd32784678a59d44bfffe22515; no voters:
I20260620 20:22:29.086050 3320 leader_election.cc:288] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:29.086199 3325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:29.087419 3325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515 [term 1 LEADER]: Becoming Leader. State: Replica: 8c50affd32784678a59d44bfffe22515, State: Running, Role: LEADER
I20260620 20:22:29.087998 3325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c50affd32784678a59d44bfffe22515" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36243 } }
I20260620 20:22:29.088155 3320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:29.090423 3327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c50affd32784678a59d44bfffe22515. Latest consensus state: current_term: 1 leader_uuid: "8c50affd32784678a59d44bfffe22515" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c50affd32784678a59d44bfffe22515" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36243 } } }
I20260620 20:22:29.090724 3327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:29.091277 3326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c50affd32784678a59d44bfffe22515" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c50affd32784678a59d44bfffe22515" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36243 } } }
I20260620 20:22:29.091634 3326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:29.093959 3332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:29.099152 3332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:29.108704 3332 catalog_manager.cc:1347] Generated new cluster ID: 90dad4f6135c4a38bcb1e5112bb18bb3
I20260620 20:22:29.108839 3332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:29.131297 3332 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:29.132647 3332 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:29.151015 3332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c50affd32784678a59d44bfffe22515: Generated new TSK 0
I20260620 20:22:29.152110 3332 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
I20260620 20:22:29.209901 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36243
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:35717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:29.549866 3346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:29.550134 3346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:29.550280 3346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:29.557358 3346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:29.557456 3346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:29.557679 3346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:22:29.567781 3346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:35717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:29.569080 3346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:29.570423 3346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:29.583316 3351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:29.583287 3354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:29.583317 3352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:29.584002 3346 server_base.cc:1034] running on GCE node
I20260620 20:22:29.584625 3346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:29.585599 3346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:29.586867 3346 hybrid_clock.cc:648] HybridClock initialized: now 1781986949586809 us; error 53 us; skew 500 ppm
I20260620 20:22:29.587175 3346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:29 dist-test-slave-zzwf krb5kdc[3240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986949, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:29.590371 3346 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:22:29.591881 3346 webserver.cc:460] Webserver started at http://127.31.235.129:33401/ using document root <none> and password file <none>
I20260620 20:22:29.592545 3346 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:29.592731 3346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:29.593083 3346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:29.595996 3346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "89b31e36f32c4bda926780b7b116fad6"
format_stamp: "Formatted at 2026-06-20 20:22:29 on dist-test-slave-zzwf"
server_key: "2e36f5b36f9d9c29bc5b00a7f7137396"
server_key_iv: "aff11076a082a8e80c4b6b5cbd01695b"
server_key_version: "encryptionkey@0"
I20260620 20:22:29.596686 3346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "89b31e36f32c4bda926780b7b116fad6"
format_stamp: "Formatted at 2026-06-20 20:22:29 on dist-test-slave-zzwf"
server_key: "2e36f5b36f9d9c29bc5b00a7f7137396"
server_key_iv: "aff11076a082a8e80c4b6b5cbd01695b"
server_key_version: "encryptionkey@0"
I20260620 20:22:29.601909 3346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 20:22:29.605340 3361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:29.606797 3346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:29.606933 3346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "89b31e36f32c4bda926780b7b116fad6"
format_stamp: "Formatted at 2026-06-20 20:22:29 on dist-test-slave-zzwf"
server_key: "2e36f5b36f9d9c29bc5b00a7f7137396"
server_key_iv: "aff11076a082a8e80c4b6b5cbd01695b"
server_key_version: "encryptionkey@0"
I20260620 20:22:29.607093 3346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:29.645840 3346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:29.650305 3346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:29.650669 3346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:29.651683 3346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:29.652930 3346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:29.653002 3346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:29.653082 3346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:29.653126 3346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:29.685791 3346 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:38151
I20260620 20:22:29.685969 3473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:38151 every 8 connection(s)
I20260620 20:22:29.687036 3346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:22:29.691480 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 3346
I20260620 20:22:29.691684 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:22:29.692489 32686 external_mini_cluster.cc:1428] Setting key 041cdf9945b7b60396712a8ddd3959bc
Jun 20 20:22:29 dist-test-slave-zzwf krb5kdc[3240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986949, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:29.709144 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.689656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:44423 (local address 127.31.235.190:36243)
0620 20:22:29.690098 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:29.690102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:29.691073 (+ 971us) server_negotiation.cc:407] Connection header received
0620 20:22:29.693611 (+ 2538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:29.693619 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:29.693697 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:29.693805 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:29.695755 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.696469 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.699015 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.699479 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.699576 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:29.703174 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:29.703179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:29.703182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:29.703214 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:29.705730 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.706506 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.706514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.706519 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.706642 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:29.707120 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:29.707128 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:29.707134 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:29.707527 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:29.707678 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:29.708433 (+ 755us) server_negotiation.cc:299] Negotiation successful
0620 20:22:29.708707 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":233,"threads_started":1}
I20260620 20:22:29.709890 3477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:29.690170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:36243 (local address 127.31.235.129:44423)
0620 20:22:29.690870 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:29.690907 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:29.693266 (+ 2359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:29.694043 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:29.694051 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:29.694760 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:29.695575 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:29.695603 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.696614 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.696622 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:29.698861 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:29.698878 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:29.699698 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:29.699707 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:29.700042 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:29.700752 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:29.700774 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:29.702962 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:29.705874 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:29.705883 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:29.705900 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:29.706360 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:29.706753 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:29.706757 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:29.706759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:29.706933 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:29.707664 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:29.707670 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:29.708025 (+ 355us) client_negotiation.cc:769] Sending connection context
0620 20:22:29.708252 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 20:22:29.708644 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":288,"threads_started":1}
WARNING: no policy specified for kudu/127.31.235.130@KRBTEST.COM; defaulting to no policy
I20260620 20:22:29.711730 3474 heartbeater.cc:347] Connected to a master server at 127.31.235.190:36243
I20260620 20:22:29.712067 3474 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:29.712862 3474 heartbeater.cc:511] Master 127.31.235.190:36243 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.130@KRBTEST.COM" created.
I20260620 20:22:29.715098 3285 ts_manager.cc:194] Registered new tserver with Master: 89b31e36f32c4bda926780b7b116fad6 (127.31.235.129:38151)
I20260620 20:22:29.716594 3285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:44423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
I20260620 20:22:29.740746 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36243
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:35717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:30.057070 3480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:30.057327 3480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:30.057482 3480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:30.064850 3480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:30.064985 3480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:30.065157 3480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:22:30.075521 3480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:35717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:30.076982 3480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:30.078392 3480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:30.091477 3488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:30.091437 3486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:30.091513 3485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:30.092088 3480 server_base.cc:1034] running on GCE node
I20260620 20:22:30.092682 3480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:30.093739 3480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:30.094973 3480 hybrid_clock.cc:648] HybridClock initialized: now 1781986950094927 us; error 45 us; skew 500 ppm
I20260620 20:22:30.095288 3480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:30 dist-test-slave-zzwf krb5kdc[3240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986950, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:30.098915 3480 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:22:30.100387 3480 webserver.cc:460] Webserver started at http://127.31.235.130:43213/ using document root <none> and password file <none>
I20260620 20:22:30.101094 3480 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:30.101235 3480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:30.101596 3480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:30.104476 3480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "98d62abea42c437199c722c04ed3d534"
format_stamp: "Formatted at 2026-06-20 20:22:30 on dist-test-slave-zzwf"
server_key: "6aa919274e11d5dc32b8b7ea9af47339"
server_key_iv: "511db97a79b76f6ee317608d77eb7975"
server_key_version: "encryptionkey@0"
I20260620 20:22:30.105247 3480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "98d62abea42c437199c722c04ed3d534"
format_stamp: "Formatted at 2026-06-20 20:22:30 on dist-test-slave-zzwf"
server_key: "6aa919274e11d5dc32b8b7ea9af47339"
server_key_iv: "511db97a79b76f6ee317608d77eb7975"
server_key_version: "encryptionkey@0"
I20260620 20:22:30.110507 3480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 20:22:30.113878 3495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:30.115149 3480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 20:22:30.115303 3480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "98d62abea42c437199c722c04ed3d534"
format_stamp: "Formatted at 2026-06-20 20:22:30 on dist-test-slave-zzwf"
server_key: "6aa919274e11d5dc32b8b7ea9af47339"
server_key_iv: "511db97a79b76f6ee317608d77eb7975"
server_key_version: "encryptionkey@0"
I20260620 20:22:30.115494 3480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:30.127449 3480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:30.131577 3480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:30.131872 3480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:30.132794 3480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:30.134145 3480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:30.134239 3480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:30.134380 3480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:30.134449 3480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:30.166787 3480 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:45665
I20260620 20:22:30.166872 3607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:45665 every 8 connection(s)
I20260620 20:22:30.168051 3480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:22:30.168497 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 3480
I20260620 20:22:30.168685 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:22:30.169332 32686 external_mini_cluster.cc:1428] Setting key 4083330d643bfff618929dc0b0de5913
Jun 20 20:22:30 dist-test-slave-zzwf krb5kdc[3240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986950, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.131@KRBTEST.COM" created.
I20260620 20:22:30.192852 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:30.170711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:41855 (local address 127.31.235.190:36243)
0620 20:22:30.170911 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:30.170918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:30.172349 (+ 1431us) server_negotiation.cc:407] Connection header received
0620 20:22:30.174221 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:30.174230 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:30.174324 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:30.174452 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:30.176037 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.176806 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.179550 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.180015 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.180123 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:30.185187 (+ 5064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:30.185198 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:30.185202 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:30.185280 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:30.188454 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:30.189821 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:30.189829 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:30.189831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:30.189997 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:30.190524 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:30.190528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:30.190531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:30.190879 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:30.191010 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:30.192231 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0620 20:22:30.192428 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:22:30.193804 3610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:30.171313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:36243 (local address 127.31.235.130:41855)
0620 20:22:30.172160 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:30.172217 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:30.173954 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:30.174693 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:30.174702 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:30.175246 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:30.175812 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:30.175829 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.176976 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.176985 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:30.179369 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:30.179393 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.180260 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.180268 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:30.180610 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:30.181375 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:30.181410 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:30.184804 (+ 3394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:30.188668 (+ 3864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:30.188682 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:30.188707 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:30.189424 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:30.190155 (+ 731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:30.190158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:30.190161 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:30.190356 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:30.191166 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:30.191173 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:30.191536 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 20:22:30.191869 (+ 333us) client_negotiation.cc:240] Negotiation successful
0620 20:22:30.192317 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":345,"threads_started":1}
I20260620 20:22:30.195472 3608 heartbeater.cc:347] Connected to a master server at 127.31.235.190:36243
I20260620 20:22:30.195804 3608 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:30.196543 3608 heartbeater.cc:511] Master 127.31.235.190:36243 requested a full tablet report, sending...
I20260620 20:22:30.198127 3285 ts_manager.cc:194] Registered new tserver with Master: 98d62abea42c437199c722c04ed3d534 (127.31.235.130:45665)
I20260620 20:22:30.199052 3285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:41855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
I20260620 20:22:30.218178 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36243
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:35717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:30.551770 3613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:30.552063 3613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:30.552242 3613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:30.559612 3613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:30.559727 3613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:30.559947 3613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:22:30.569805 3613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:35717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:30.571197 3613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:30.572594 3613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:30.586558 3619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:30.586511 3621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:30.586592 3618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:30.587170 3613 server_base.cc:1034] running on GCE node
I20260620 20:22:30.587867 3613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:30.588876 3613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:30.590126 3613 hybrid_clock.cc:648] HybridClock initialized: now 1781986950590075 us; error 36 us; skew 500 ppm
I20260620 20:22:30.590497 3613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:30 dist-test-slave-zzwf krb5kdc[3240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986950, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:30.594175 3613 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:22:30.595592 3613 webserver.cc:460] Webserver started at http://127.31.235.131:38283/ using document root <none> and password file <none>
I20260620 20:22:30.596297 3613 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:30.596400 3613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:30.596705 3613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:30.599893 3613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "a66516ad73a14bcba90accb969729391"
format_stamp: "Formatted at 2026-06-20 20:22:30 on dist-test-slave-zzwf"
server_key: "a3b35e213177eef1d6d8f48815de220d"
server_key_iv: "fdcf22b4698e931056a5ce338e97b396"
server_key_version: "encryptionkey@0"
I20260620 20:22:30.600708 3613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "a66516ad73a14bcba90accb969729391"
format_stamp: "Formatted at 2026-06-20 20:22:30 on dist-test-slave-zzwf"
server_key: "a3b35e213177eef1d6d8f48815de220d"
server_key_iv: "fdcf22b4698e931056a5ce338e97b396"
server_key_version: "encryptionkey@0"
I20260620 20:22:30.606370 3613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 20:22:30.610436 3628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:30.611991 3613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:22:30.612162 3613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "a66516ad73a14bcba90accb969729391"
format_stamp: "Formatted at 2026-06-20 20:22:30 on dist-test-slave-zzwf"
server_key: "a3b35e213177eef1d6d8f48815de220d"
server_key_iv: "fdcf22b4698e931056a5ce338e97b396"
server_key_version: "encryptionkey@0"
I20260620 20:22:30.612389 3613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:30.633379 3613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:30.637866 3613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:30.638200 3613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:30.639204 3613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:30.640573 3613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:30.640676 3613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:30.640818 3613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:30.640887 3613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:30.673388 3613 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:35497
I20260620 20:22:30.673470 3742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:35497 every 8 connection(s)
I20260620 20:22:30.674732 3613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:22:30.676986 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 3613
I20260620 20:22:30.677158 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:22:30.677649 32686 external_mini_cluster.cc:1428] Setting key 8999740b1b5dc4dbfcf2dea23ff40827
Jun 20 20:22:30 dist-test-slave-zzwf krb5kdc[3240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986950, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:30.696692 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:30.677215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:52183 (local address 127.31.235.190:36243)
0620 20:22:30.677697 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:30.677704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:30.678791 (+ 1087us) server_negotiation.cc:407] Connection header received
0620 20:22:30.680586 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:30.680589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:30.680652 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:30.680732 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:30.682291 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.683018 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.684550 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.684987 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.685096 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:30.690055 (+ 4959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:30.690061 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:30.690064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:30.690100 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:30.693670 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:30.694525 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:30.694534 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:30.694540 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:30.694693 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:30.695137 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:30.695141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:30.695143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:30.695417 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:30.695532 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:30.696283 (+ 751us) server_negotiation.cc:299] Negotiation successful
0620 20:22:30.696449 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344}
I20260620 20:22:30.697883 3745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:30.677815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:36243 (local address 127.31.235.131:52183)
0620 20:22:30.678631 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:30.678682 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:30.680331 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:30.680948 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:30.680956 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:30.681566 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:30.682137 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:30.682154 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.683145 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.683149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:30.684384 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:30.684395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.685306 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.685317 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:30.685688 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:30.686659 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:30.686702 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:30.689771 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:30.693883 (+ 4112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:30.693891 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:30.693906 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:30.694359 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:30.694803 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:30.694843 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:30.694845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:30.695035 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:30.695545 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:30.695552 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:30.695944 (+ 392us) client_negotiation.cc:769] Sending connection context
0620 20:22:30.696187 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 20:22:30.696608 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":332,"threads_started":1}
I20260620 20:22:30.699450 3743 heartbeater.cc:347] Connected to a master server at 127.31.235.190:36243
I20260620 20:22:30.699775 3743 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:30.700546 3743 heartbeater.cc:511] Master 127.31.235.190:36243 requested a full tablet report, sending...
I20260620 20:22:30.702221 3285 ts_manager.cc:194] Registered new tserver with Master: a66516ad73a14bcba90accb969729391 (127.31.235.131:35497)
I20260620 20:22:30.703030 3285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:52183
I20260620 20:22:30.705531 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:22:30.720220 3474 heartbeater.cc:503] Master 127.31.235.190:36243 was elected leader, sending a full tablet report...
Jun 20 20:22:30 dist-test-slave-zzwf krb5kdc[3240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986950, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 20:22:30 dist-test-slave-zzwf krb5kdc[3240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986950, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:30.760063 3476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:30.741585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44450 (local address 127.31.235.190:36243)
0620 20:22:30.741797 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:30.741802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:30.742266 (+ 464us) server_negotiation.cc:407] Connection header received
0620 20:22:30.742433 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:30.742437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:30.742508 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:30.742614 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:30.743956 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.744690 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.746025 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.746475 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.746578 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:30.755015 (+ 8437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:30.755021 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:30.755025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:30.755064 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:30.757702 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:30.758399 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:30.758403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:30.758406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:30.758532 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:30.758909 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:30.758912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:30.758914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:30.759238 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:30.759412 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:30.759675 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 20:22:30.759839 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 20:22:30.760600 3285 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44450
I20260620 20:22:30.762380 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 3346
Jun 20 20:22:30 dist-test-slave-zzwf krb5kdc[3240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986950, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:22:30.777788 3754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:30.763303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52208 (local address 127.31.235.129:38151)
0620 20:22:30.763872 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:30.763879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:30.763904 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:22:30.763958 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:30.763965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:30.764170 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:30.764291 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:30.765385 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.766426 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.767670 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.768201 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.768295 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:30.772404 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:30.772416 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:30.772441 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:30.772472 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:30.775364 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:30.776036 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:30.776042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:30.776045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:30.776155 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:30.776517 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:30.776520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:30.776521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:30.776829 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:30.777048 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:30.777251 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 20:22:30.777429 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":270,"threads_started":1}
W20260620 20:22:30.778988 3469 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52208
W20260620 20:22:30.780097 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 3346: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:22:30.780305 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 3346
I20260620 20:22:30.795934 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 3480
Jun 20 20:22:30 dist-test-slave-zzwf krb5kdc[3240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986950, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.235.130@KRBTEST.COM
I20260620 20:22:30.812458 3755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:30.796800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45554 (local address 127.31.235.130:45665)
0620 20:22:30.797407 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:30.797414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:30.797441 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:22:30.797534 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:30.797546 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:30.797794 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:30.797975 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:30.799246 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.800270 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.801468 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.802070 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.802176 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:30.806562 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:30.806573 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:30.806588 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:30.806622 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:30.809702 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:30.810376 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:30.810384 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:30.810386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:30.810504 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:30.810882 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:30.810887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:30.810890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:30.811219 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:30.811536 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:30.811839 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 20:22:30.812104 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":332,"threads_started":1}
W20260620 20:22:30.813781 3603 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45554
W20260620 20:22:30.814844 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 3480: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:22:30.814988 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 3480
I20260620 20:22:30.831666 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 3613
Jun 20 20:22:30 dist-test-slave-zzwf krb5kdc[3240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986950, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.235.131@KRBTEST.COM
I20260620 20:22:30.847939 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:30.832620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44564 (local address 127.31.235.131:35497)
0620 20:22:30.833235 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:30.833242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:30.833269 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:22:30.833333 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:30.833342 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:30.833618 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:30.833775 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:30.835098 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.836143 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.837375 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:30.838044 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:30.838180 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:30.842363 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:30.842374 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:30.842388 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:30.842438 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:30.845161 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:30.845876 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:30.845882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:30.845885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:30.845998 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:30.846344 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:30.846351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:30.846354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:30.846683 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:30.846994 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:30.847325 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 20:22:30.847596 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":321,"threads_started":1}
W20260620 20:22:30.849264 3738 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44564
W20260620 20:22:30.850384 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 3613: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:22:30.850564 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 3613
I20260620 20:22:30.865823 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 3253
I20260620 20:22:30.867527 3315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:30.989670 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 3253
2026-06-20T20:22:31Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3627 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:31 dist-test-slave-zzwf krb5kdc[3760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:31 dist-test-slave-zzwf krb5kdc[3760](info): set up 2 sockets
Jun 20 20:22:31 dist-test-slave-zzwf krb5kdc[3760](info): commencing operation
krb5kdc: starting...
W20260620 20:22:32.094242 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:22:32 dist-test-slave-zzwf krb5kdc[3760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:22:32.301222 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:37489
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:39953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:37489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:32.618065 3773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:32.618420 3773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:32.618580 3773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:32.625741 3773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:32.625913 3773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:32.626017 3773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:32.626118 3773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:32.626200 3773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:32.635869 3773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:39953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:37489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:37489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:32.637203 3773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:32.638473 3773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:32.650178 3778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:32.650276 3781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:32.650240 3779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:32.650527 3773 server_base.cc:1034] running on GCE node
I20260620 20:22:32.651364 3773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:32.652781 3773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:32.654031 3773 hybrid_clock.cc:648] HybridClock initialized: now 1781986952653976 us; error 66 us; skew 500 ppm
I20260620 20:22:32.654384 3773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:32 dist-test-slave-zzwf krb5kdc[3760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986952, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:32.657569 3773 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:22:32.658995 3773 webserver.cc:460] Webserver started at http://127.31.235.190:44681/ using document root <none> and password file <none>
I20260620 20:22:32.659672 3773 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:32.659818 3773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:32.660130 3773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:32.663178 3773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "d3046ab25b4f4a52af4e0ad03b9d4cf8"
format_stamp: "Formatted at 2026-06-20 20:22:32 on dist-test-slave-zzwf"
server_key: "05059100419b5ac2f140f7eedb6cad80"
server_key_iv: "f3b28b030bf5f2e981c2d707ba5281a3"
server_key_version: "encryptionkey@0"
I20260620 20:22:32.663944 3773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "d3046ab25b4f4a52af4e0ad03b9d4cf8"
format_stamp: "Formatted at 2026-06-20 20:22:32 on dist-test-slave-zzwf"
server_key: "05059100419b5ac2f140f7eedb6cad80"
server_key_iv: "f3b28b030bf5f2e981c2d707ba5281a3"
server_key_version: "encryptionkey@0"
I20260620 20:22:32.668730 3773 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 20:22:32.672147 3788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:32.673390 3773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:22:32.673604 3773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "d3046ab25b4f4a52af4e0ad03b9d4cf8"
format_stamp: "Formatted at 2026-06-20 20:22:32 on dist-test-slave-zzwf"
server_key: "05059100419b5ac2f140f7eedb6cad80"
server_key_iv: "f3b28b030bf5f2e981c2d707ba5281a3"
server_key_version: "encryptionkey@0"
I20260620 20:22:32.673794 3773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:32.701313 3773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:32.705710 3773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:32.706004 3773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:32.724319 3773 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:37489
I20260620 20:22:32.724323 3840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:37489 every 8 connection(s)
I20260620 20:22:32.725812 3773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:32.729619 3841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:32.730378 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 3773
I20260620 20:22:32.730587 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:22:32.731211 32686 external_mini_cluster.cc:1428] Setting key 2f2fbb2a6bb170e8db6addc4f14687aa
I20260620 20:22:32.744377 3841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8: Bootstrap starting.
Jun 20 20:22:32 dist-test-slave-zzwf krb5kdc[3760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:32.748458 3841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:32.749902 3841 log.cc:824] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:32.753337 3841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8: No bootstrap required, opened a new log
I20260620 20:22:32.756461 3844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:32.733287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53808 (local address 127.31.235.190:37489)
0620 20:22:32.734208 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:32.734225 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:32.734276 (+ 51us) server_negotiation.cc:407] Connection header received
0620 20:22:32.736260 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:32.736292 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:32.736842 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:32.737327 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:32.738784 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:32.740780 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:32.742150 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:32.743232 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:32.743406 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:32.748036 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:32.748047 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:32.748068 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:32.748120 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:32.751893 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:32.752594 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:32.752602 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:32.752610 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:32.752817 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:32.753175 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:32.753179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:32.753182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:32.753917 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:32.754235 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:32.754552 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 20:22:32.754993 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":401,"threads_started":1}
I20260620 20:22:32.762742 3841 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3046ab25b4f4a52af4e0ad03b9d4cf8" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 37489 } }
I20260620 20:22:32.763104 3841 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:32.763167 3841 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3046ab25b4f4a52af4e0ad03b9d4cf8, State: Initialized, Role: FOLLOWER
I20260620 20:22:32.763736 3841 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3046ab25b4f4a52af4e0ad03b9d4cf8" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 37489 } }
I20260620 20:22:32.763935 3841 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:32.764035 3841 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:32.764158 3841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:32.766307 3841 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3046ab25b4f4a52af4e0ad03b9d4cf8" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 37489 } }
I20260620 20:22:32.766767 3841 leader_election.cc:302] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3046ab25b4f4a52af4e0ad03b9d4cf8; no voters:
I20260620 20:22:32.767325 3841 leader_election.cc:288] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:32.767506 3846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:32.768697 3846 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8 [term 1 LEADER]: Becoming Leader. State: Replica: d3046ab25b4f4a52af4e0ad03b9d4cf8, State: Running, Role: LEADER
I20260620 20:22:32.769299 3846 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3046ab25b4f4a52af4e0ad03b9d4cf8" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 37489 } }
I20260620 20:22:32.769610 3841 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:32.772512 3848 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3046ab25b4f4a52af4e0ad03b9d4cf8. Latest consensus state: current_term: 1 leader_uuid: "d3046ab25b4f4a52af4e0ad03b9d4cf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3046ab25b4f4a52af4e0ad03b9d4cf8" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 37489 } } }
I20260620 20:22:32.772500 3847 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3046ab25b4f4a52af4e0ad03b9d4cf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3046ab25b4f4a52af4e0ad03b9d4cf8" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 37489 } } }
I20260620 20:22:32.772953 3847 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:32.772953 3848 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:32.777679 3856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:32.782032 3856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:32.790300 3856 catalog_manager.cc:1347] Generated new cluster ID: 45dbc5e23418466faef1dcf43f4e637d
I20260620 20:22:32.790441 3856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:32.802717 3856 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:32.804080 3856 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:32.820461 3856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3046ab25b4f4a52af4e0ad03b9d4cf8: Generated new TSK 0
I20260620 20:22:32.821404 3856 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
I20260620 20:22:32.883379 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:37489
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:39953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:33.218578 3867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:33.218858 3867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:33.218948 3867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:33.226118 3867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:33.226234 3867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:33.226356 3867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:22:33.236172 3867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:39953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:37489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:33.237409 3867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:33.238735 3867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:33.251818 3873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:33.251884 3872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:33.251860 3875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:33.252579 3867 server_base.cc:1034] running on GCE node
I20260620 20:22:33.253194 3867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:33.254184 3867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:33.255385 3867 hybrid_clock.cc:648] HybridClock initialized: now 1781986953255330 us; error 50 us; skew 500 ppm
I20260620 20:22:33.255743 3867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:33 dist-test-slave-zzwf krb5kdc[3760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986953, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:33.259099 3867 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:22:33.260637 3867 webserver.cc:460] Webserver started at http://127.31.235.129:36861/ using document root <none> and password file <none>
I20260620 20:22:33.261312 3867 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:33.261413 3867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:33.261736 3867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:33.264683 3867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "e131d411c3654162ad1a9bffea92a83d"
format_stamp: "Formatted at 2026-06-20 20:22:33 on dist-test-slave-zzwf"
server_key: "8448d454c414663d6a335d6e63b973f2"
server_key_iv: "82aa8524c7afc2641e7cbc3dd77fc487"
server_key_version: "encryptionkey@0"
I20260620 20:22:33.265480 3867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "e131d411c3654162ad1a9bffea92a83d"
format_stamp: "Formatted at 2026-06-20 20:22:33 on dist-test-slave-zzwf"
server_key: "8448d454c414663d6a335d6e63b973f2"
server_key_iv: "82aa8524c7afc2641e7cbc3dd77fc487"
server_key_version: "encryptionkey@0"
I20260620 20:22:33.270723 3867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:22:33.274616 3882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:33.275905 3867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:22:33.276075 3867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "e131d411c3654162ad1a9bffea92a83d"
format_stamp: "Formatted at 2026-06-20 20:22:33 on dist-test-slave-zzwf"
server_key: "8448d454c414663d6a335d6e63b973f2"
server_key_iv: "82aa8524c7afc2641e7cbc3dd77fc487"
server_key_version: "encryptionkey@0"
I20260620 20:22:33.276289 3867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:33.302632 3867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:33.307142 3867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:33.307446 3867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:33.308435 3867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:33.309880 3867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:33.309983 3867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:33.310134 3867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:33.310206 3867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:33.343248 3867 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:41451
I20260620 20:22:33.343336 3995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:41451 every 8 connection(s)
I20260620 20:22:33.344521 3867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:22:33.353751 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 3867
I20260620 20:22:33.353958 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:22:33.354712 32686 external_mini_cluster.cc:1428] Setting key ae62fe7eee3e4c1740197744499359d8
Jun 20 20:22:33 dist-test-slave-zzwf krb5kdc[3760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986953, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:33.364953 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:33.346988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:42307 (local address 127.31.235.190:37489)
0620 20:22:33.347434 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:33.347438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:33.348247 (+ 809us) server_negotiation.cc:407] Connection header received
0620 20:22:33.350085 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:33.350090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:33.350178 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:33.350291 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:33.351743 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:33.352478 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:33.354623 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:33.355113 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:33.355232 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:33.359215 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:33.359220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:33.359223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:33.359253 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:33.361772 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:33.362490 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:33.362498 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:33.362506 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:33.362617 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:33.363130 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:33.363134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:33.363136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:33.363404 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:33.363515 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:33.364493 (+ 978us) server_negotiation.cc:299] Negotiation successful
0620 20:22:33.364677 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":244,"threads_started":1}
I20260620 20:22:33.366508 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:33.347442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:37489 (local address 127.31.235.129:42307)
0620 20:22:33.348092 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:33.348134 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:33.349830 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:33.350527 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:33.350535 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:33.351032 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:33.351576 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:33.351593 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:33.352643 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:33.352647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:33.354420 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:33.354443 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:33.355352 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:33.355358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:33.355607 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:33.356455 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:33.356498 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:33.358996 (+ 2498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:33.361914 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:33.361923 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:33.361942 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:33.362360 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:33.362723 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:33.362729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:33.362733 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:33.362998 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:33.363537 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:33.363547 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:33.364027 (+ 480us) client_negotiation.cc:769] Sending connection context
0620 20:22:33.364330 (+ 303us) client_negotiation.cc:240] Negotiation successful
0620 20:22:33.364913 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":249,"threads_started":1}
I20260620 20:22:33.368175 3996 heartbeater.cc:347] Connected to a master server at 127.31.235.190:37489
I20260620 20:22:33.368500 3996 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:33.369365 3996 heartbeater.cc:511] Master 127.31.235.190:37489 requested a full tablet report, sending...
I20260620 20:22:33.371783 3805 ts_manager.cc:194] Registered new tserver with Master: e131d411c3654162ad1a9bffea92a83d (127.31.235.129:41451)
I20260620 20:22:33.373426 3805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:42307
WARNING: no policy specified for kudu/127.31.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
I20260620 20:22:33.401651 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:37489
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:39953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:33.717126 4002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:33.717453 4002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:33.717620 4002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:33.724759 4002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:33.724918 4002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:33.725106 4002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:22:33.735127 4002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:39953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:37489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:33.736377 4002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:33.737731 4002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:33.751274 4008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:33.751605 4010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:33.751958 4007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:33.752826 4002 server_base.cc:1034] running on GCE node
I20260620 20:22:33.753437 4002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:33.754436 4002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:33.755703 4002 hybrid_clock.cc:648] HybridClock initialized: now 1781986953755632 us; error 43 us; skew 500 ppm
I20260620 20:22:33.756047 4002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:33 dist-test-slave-zzwf krb5kdc[3760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986953, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:33.759217 4002 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:22:33.760710 4002 webserver.cc:460] Webserver started at http://127.31.235.130:37243/ using document root <none> and password file <none>
I20260620 20:22:33.761397 4002 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:33.761555 4002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:33.761861 4002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:33.765004 4002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "17bea8dae4cd49a699d2e7ff5121224c"
format_stamp: "Formatted at 2026-06-20 20:22:33 on dist-test-slave-zzwf"
server_key: "a4dae7b7dcfd103af6950dc50ed1bfe9"
server_key_iv: "4d1ab4d03778d9faca93a5205c4f071c"
server_key_version: "encryptionkey@0"
I20260620 20:22:33.765941 4002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "17bea8dae4cd49a699d2e7ff5121224c"
format_stamp: "Formatted at 2026-06-20 20:22:33 on dist-test-slave-zzwf"
server_key: "a4dae7b7dcfd103af6950dc50ed1bfe9"
server_key_iv: "4d1ab4d03778d9faca93a5205c4f071c"
server_key_version: "encryptionkey@0"
I20260620 20:22:33.771667 4002 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260620 20:22:33.775421 4017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:33.776827 4002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:22:33.777021 4002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "17bea8dae4cd49a699d2e7ff5121224c"
format_stamp: "Formatted at 2026-06-20 20:22:33 on dist-test-slave-zzwf"
server_key: "a4dae7b7dcfd103af6950dc50ed1bfe9"
server_key_iv: "4d1ab4d03778d9faca93a5205c4f071c"
server_key_version: "encryptionkey@0"
I20260620 20:22:33.777232 4002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:33.842803 4002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:33.847373 4002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:33.847694 4002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:33.848769 4002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:33.850612 4002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:33.850723 4002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:33.850878 4002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:33.850965 4002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:33.886480 4002 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:46471
I20260620 20:22:33.886579 4130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:46471 every 8 connection(s)
I20260620 20:22:33.887861 4002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:22:33.891281 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 4002
I20260620 20:22:33.891484 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:22:33.891907 32686 external_mini_cluster.cc:1428] Setting key 8ef0cd9df6d73a10dcbf27ef24fb95c3
Jun 20 20:22:33 dist-test-slave-zzwf krb5kdc[3760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986953, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:33.909837 4133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:33.890483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:57887 (local address 127.31.235.190:37489)
0620 20:22:33.890967 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:33.890973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:33.892077 (+ 1104us) server_negotiation.cc:407] Connection header received
0620 20:22:33.894377 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:33.894383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:33.894468 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:33.894601 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:33.896346 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:33.897632 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:33.899373 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:33.899912 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:33.900064 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:33.904284 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:33.904289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:33.904292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:33.904322 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:33.906899 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:33.907639 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:33.907647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:33.907653 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:33.907776 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:33.908195 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:33.908200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:33.908203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:33.908449 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:33.908595 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:33.909248 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 20:22:33.909450 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":213,"threads_started":1}
I20260620 20:22:33.910821 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:33.891066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:37489 (local address 127.31.235.130:57887)
0620 20:22:33.891894 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:33.891953 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:33.894095 (+ 2142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:33.894892 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:33.894902 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:33.895506 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:33.896122 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:33.896156 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:33.897812 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:33.897817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:33.899212 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:33.899224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:33.900180 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:33.900189 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:33.900437 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:33.901198 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:33.901240 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:33.904049 (+ 2809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:33.907072 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:33.907081 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:33.907099 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:33.907521 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:33.907890 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:33.907895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:33.907899 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:33.908077 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:33.908568 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:33.908576 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:33.908912 (+ 336us) client_negotiation.cc:769] Sending connection context
0620 20:22:33.909185 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 20:22:33.909736 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":368,"threads_started":1}
WARNING: no policy specified for kudu/127.31.235.131@KRBTEST.COM; defaulting to no policy
I20260620 20:22:33.912266 4131 heartbeater.cc:347] Connected to a master server at 127.31.235.190:37489
I20260620 20:22:33.912523 4131 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:33.913239 4131 heartbeater.cc:511] Master 127.31.235.190:37489 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.131@KRBTEST.COM" created.
I20260620 20:22:33.914834 3805 ts_manager.cc:194] Registered new tserver with Master: 17bea8dae4cd49a699d2e7ff5121224c (127.31.235.130:46471)
I20260620 20:22:33.915628 3805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:57887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
I20260620 20:22:33.939206 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:37489
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:39953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:34.271776 4137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:34.272079 4137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:34.272275 4137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:34.283655 4137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:34.283828 4137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:34.284111 4137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:22:34.295506 4137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:39953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:37489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:34.296873 4137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:34.298326 4137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:34.312024 4142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:34.312112 4145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:34.312080 4143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:34.313246 4137 server_base.cc:1034] running on GCE node
I20260620 20:22:34.314038 4137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:34.315043 4137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:34.316288 4137 hybrid_clock.cc:648] HybridClock initialized: now 1781986954316200 us; error 60 us; skew 500 ppm
I20260620 20:22:34.316706 4137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:34 dist-test-slave-zzwf krb5kdc[3760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986954, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:34.324074 4137 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:22:34.325660 4137 webserver.cc:460] Webserver started at http://127.31.235.131:41781/ using document root <none> and password file <none>
I20260620 20:22:34.326303 4137 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:34.326396 4137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:34.326725 4137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:34.329545 4137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "3cea3b54c6fe4411a0802f4ea673ce00"
format_stamp: "Formatted at 2026-06-20 20:22:34 on dist-test-slave-zzwf"
server_key: "fc2bbb09931cd0a66edee56225a749fc"
server_key_iv: "7a2bc683bae2626d193ec7a3b613423a"
server_key_version: "encryptionkey@0"
I20260620 20:22:34.330298 4137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "3cea3b54c6fe4411a0802f4ea673ce00"
format_stamp: "Formatted at 2026-06-20 20:22:34 on dist-test-slave-zzwf"
server_key: "fc2bbb09931cd0a66edee56225a749fc"
server_key_iv: "7a2bc683bae2626d193ec7a3b613423a"
server_key_version: "encryptionkey@0"
I20260620 20:22:34.335234 4137 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:22:34.338579 4152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:34.339855 4137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:22:34.339972 4137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "3cea3b54c6fe4411a0802f4ea673ce00"
format_stamp: "Formatted at 2026-06-20 20:22:34 on dist-test-slave-zzwf"
server_key: "fc2bbb09931cd0a66edee56225a749fc"
server_key_iv: "7a2bc683bae2626d193ec7a3b613423a"
server_key_version: "encryptionkey@0"
I20260620 20:22:34.340135 4137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:34.355933 4137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:34.360117 4137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:34.360365 4137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:34.361243 4137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:34.362591 4137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:34.362663 4137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:34.362743 4137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:34.362792 4137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:34.376739 3996 heartbeater.cc:503] Master 127.31.235.190:37489 was elected leader, sending a full tablet report...
I20260620 20:22:34.396766 4137 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:39113
I20260620 20:22:34.396795 4265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:39113 every 8 connection(s)
I20260620 20:22:34.398072 4137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:22:34.408165 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 4137
I20260620 20:22:34.408383 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:22:34.408812 32686 external_mini_cluster.cc:1428] Setting key d6019123b936fa8c44f4cf480f8d63d6
Jun 20 20:22:34 dist-test-slave-zzwf krb5kdc[3760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986954, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:34.417060 4133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:34.400510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:47243 (local address 127.31.235.190:37489)
0620 20:22:34.400678 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:34.400682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:34.401841 (+ 1159us) server_negotiation.cc:407] Connection header received
0620 20:22:34.403795 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:34.403800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:34.403909 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:34.404020 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:34.405473 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:34.406229 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:34.407744 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:34.408149 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:34.408283 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:34.412041 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:34.412045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:34.412048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:34.412078 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:34.414561 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:34.415234 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:34.415237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:34.415239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:34.415345 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:34.415693 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:34.415696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:34.415698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:34.415952 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:34.416059 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:34.416660 (+ 601us) server_negotiation.cc:299] Negotiation successful
0620 20:22:34.416815 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:22:34.418191 4268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:34.400951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:37489 (local address 127.31.235.131:47243)
0620 20:22:34.401689 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:34.401732 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:34.403515 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:34.404239 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:34.404247 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:34.404754 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:34.405306 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:34.405322 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:34.406376 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:34.406391 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:34.407587 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:34.407599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:34.408367 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:34.408374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:34.408660 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:34.409363 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:34.409396 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:34.411830 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:34.414689 (+ 2859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:34.414695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:34.414708 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:34.415134 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:34.415432 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:34.415435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:34.415437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:34.415606 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:34.416054 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:34.416060 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:34.416363 (+ 303us) client_negotiation.cc:769] Sending connection context
0620 20:22:34.416580 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 20:22:34.416940 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":262,"threads_started":1}
I20260620 20:22:34.419490 4266 heartbeater.cc:347] Connected to a master server at 127.31.235.190:37489
I20260620 20:22:34.419754 4266 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:34.420392 4266 heartbeater.cc:511] Master 127.31.235.190:37489 requested a full tablet report, sending...
I20260620 20:22:34.421795 3805 ts_manager.cc:194] Registered new tserver with Master: 3cea3b54c6fe4411a0802f4ea673ce00 (127.31.235.131:39113)
I20260620 20:22:34.422724 3805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:47243
I20260620 20:22:34.424307 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:22:34 dist-test-slave-zzwf krb5kdc[3760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:22:34 dist-test-slave-zzwf krb5kdc[3760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:22:34.473268 4273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:34.457966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44524 (local address 127.31.235.129:41451)
0620 20:22:34.458647 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:34.458653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:34.458678 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:22:34.458735 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:34.458741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:34.458970 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:34.459142 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:34.460563 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:34.461693 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:34.462858 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:34.463412 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:34.463551 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:34.467406 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:34.467417 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:34.467432 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:34.467471 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:34.470340 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:34.471087 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:34.471095 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:34.471097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:34.471219 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:34.471583 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:34.471589 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:34.471592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:34.471923 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:34.472255 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:34.472580 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 20:22:34.472843 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":317,"threads_started":1}
Jun 20 20:22:34 dist-test-slave-zzwf krb5kdc[3760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:34.492192 4133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:34.477692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53820 (local address 127.31.235.190:37489)
0620 20:22:34.477914 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:34.477920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:34.478111 (+ 191us) server_negotiation.cc:407] Connection header received
0620 20:22:34.478304 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:34.478308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:34.478407 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:34.478533 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:34.480089 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:34.480963 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:34.482339 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:34.482741 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:34.482870 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:34.486776 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:34.486782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:34.486786 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:34.486827 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:34.489333 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:34.490218 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:34.490222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:34.490224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:34.490348 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:34.490720 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:34.490723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:34.490726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:34.491021 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:34.491208 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:34.491564 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 20:22:34.491803 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 20:22:34.492910 3805 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53820
I20260620 20:22:34.494102 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 3867
I20260620 20:22:34.508524 4273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:34.495329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44534 (local address 127.31.235.129:41451)
0620 20:22:34.495521 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:34.495528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:34.495547 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:22:34.495728 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:34.495735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:34.495830 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:34.495969 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:34.497144 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:34.497900 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:34.499145 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:34.499558 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:34.499695 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:34.502047 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:34.502052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:34.502055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:34.502087 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:34.505576 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:34.506542 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:34.506546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:34.506549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:34.506683 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:34.507133 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:34.507137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:34.507139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:34.507429 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:34.507610 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:34.508049 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 20:22:34.508237 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:22:34.509197 3991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:34.593753 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 3867
I20260620 20:22:34.611316 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 4002
Jun 20 20:22:34 dist-test-slave-zzwf krb5kdc[3760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.130@KRBTEST.COM
I20260620 20:22:34.626770 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:34.612168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51948 (local address 127.31.235.130:46471)
0620 20:22:34.612671 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:34.612678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:34.612703 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:22:34.612760 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:34.612766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:34.612973 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:34.613140 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:34.614352 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:34.615313 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:34.616544 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:34.617086 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:34.617228 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:34.621230 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:34.621241 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:34.621255 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:34.621287 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:34.624010 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:34.624658 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:34.624664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:34.624667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:34.624780 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:34.625141 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:34.625148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:34.625151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:34.625479 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:34.625833 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:34.626163 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 20:22:34.626428 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":248,"threads_started":1}
I20260620 20:22:34.628151 4126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:34.710978 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 4002
I20260620 20:22:34.727062 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 4137
Jun 20 20:22:34 dist-test-slave-zzwf krb5kdc[3760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.131@KRBTEST.COM
I20260620 20:22:34.745229 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:34.728065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33218 (local address 127.31.235.131:39113)
0620 20:22:34.728869 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:34.728879 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:34.728917 (+ 38us) server_negotiation.cc:407] Connection header received
0620 20:22:34.728990 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:34.729000 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:34.729308 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:34.729480 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:34.730693 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:34.732517 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:34.733994 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:34.734594 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:34.734701 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:34.739024 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:34.739036 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:34.739052 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:34.739088 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:34.742212 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:34.742935 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:34.742943 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:34.742945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:34.743064 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:34.743481 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:34.743486 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:34.743488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:34.743877 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:34.744210 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:34.744530 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 20:22:34.744815 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":428,"threads_started":1}
I20260620 20:22:34.746594 4261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:34.829608 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 4137
I20260620 20:22:34.845655 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 3773
I20260620 20:22:34.846913 3836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:34.948444 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 3773
2026-06-20T20:22:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3958 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:35 dist-test-slave-zzwf krb5kdc[4285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:35 dist-test-slave-zzwf krb5kdc[4285](info): set up 2 sockets
Jun 20 20:22:35 dist-test-slave-zzwf krb5kdc[4285](info): commencing operation
krb5kdc: starting...
W20260620 20:22:36.042407 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:22:36 dist-test-slave-zzwf krb5kdc[4285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:22:36.247321 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:33455
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:33455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:36.577033 4298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:36.577381 4298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:36.577613 4298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:36.584926 4298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:36.585084 4298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:36.585201 4298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:36.585314 4298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:36.585398 4298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:36.595430 4298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:33455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:33455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:36.596860 4298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:36.598286 4298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:36.610246 4304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:36.610388 4303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:36.610409 4306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:36.610998 4298 server_base.cc:1034] running on GCE node
I20260620 20:22:36.611723 4298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:36.613298 4298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:36.614560 4298 hybrid_clock.cc:648] HybridClock initialized: now 1781986956614484 us; error 70 us; skew 500 ppm
I20260620 20:22:36.614979 4298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:36 dist-test-slave-zzwf krb5kdc[4285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986956, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:36.618045 4298 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:22:36.619207 4298 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:36.619292 4298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:36.619545 4298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:36.622450 4298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "68b5153217dc4fd8b23f292836dde54f"
format_stamp: "Formatted at 2026-06-20 20:22:36 on dist-test-slave-zzwf"
server_key: "99cd6d09e655cab95b17826e904f5910"
server_key_iv: "99390fd980217a0c2dac20c22c56209b"
server_key_version: "encryptionkey@0"
I20260620 20:22:36.623260 4298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "68b5153217dc4fd8b23f292836dde54f"
format_stamp: "Formatted at 2026-06-20 20:22:36 on dist-test-slave-zzwf"
server_key: "99cd6d09e655cab95b17826e904f5910"
server_key_iv: "99390fd980217a0c2dac20c22c56209b"
server_key_version: "encryptionkey@0"
I20260620 20:22:36.628204 4298 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:22:36.631618 4312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:36.632985 4298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 20:22:36.633131 4298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "68b5153217dc4fd8b23f292836dde54f"
format_stamp: "Formatted at 2026-06-20 20:22:36 on dist-test-slave-zzwf"
server_key: "99cd6d09e655cab95b17826e904f5910"
server_key_iv: "99390fd980217a0c2dac20c22c56209b"
server_key_version: "encryptionkey@0"
I20260620 20:22:36.633314 4298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:36.647866 4298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:36.652019 4298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:36.652243 4298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:36.670419 4298 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:33455
I20260620 20:22:36.670444 4364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:33455 every 8 connection(s)
I20260620 20:22:36.671324 4298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:36.675197 4365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:36.679499 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 4298
I20260620 20:22:36.679716 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:22:36.680502 32686 external_mini_cluster.cc:1428] Setting key b3e74723cc7fe093713da844ba65733a
I20260620 20:22:36.687712 4365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f: Bootstrap starting.
I20260620 20:22:36.691413 4365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:36.692579 4365 log.cc:824] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:36.695995 4365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f: No bootstrap required, opened a new log
Jun 20 20:22:36 dist-test-slave-zzwf krb5kdc[4285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:36.706621 4368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:36.682696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34688 (local address 127.31.235.190:33455)
0620 20:22:36.683721 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:36.683738 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:36.683792 (+ 54us) server_negotiation.cc:407] Connection header received
0620 20:22:36.686137 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:36.686162 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:36.686571 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:36.686900 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:36.688831 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:36.690307 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:36.692059 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:36.692782 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:36.692912 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:36.697638 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:36.697648 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:36.697673 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:36.697724 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:36.701727 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:36.702472 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:36.702482 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:36.702492 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:36.702716 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:36.703165 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:36.703176 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:36.703179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:36.703967 (+ 788us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:36.704263 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:36.704690 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 20:22:36.705175 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":496,"threads_started":1}
I20260620 20:22:36.706804 4365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b5153217dc4fd8b23f292836dde54f" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33455 } }
I20260620 20:22:36.707239 4365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:36.707363 4365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68b5153217dc4fd8b23f292836dde54f, State: Initialized, Role: FOLLOWER
I20260620 20:22:36.707917 4365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b5153217dc4fd8b23f292836dde54f" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33455 } }
I20260620 20:22:36.708163 4365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:36.708308 4365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:36.708498 4365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:36.711227 4365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b5153217dc4fd8b23f292836dde54f" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33455 } }
I20260620 20:22:36.711814 4365 leader_election.cc:302] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68b5153217dc4fd8b23f292836dde54f; no voters:
I20260620 20:22:36.712388 4365 leader_election.cc:288] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:36.712568 4370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:36.713419 4370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f [term 1 LEADER]: Becoming Leader. State: Replica: 68b5153217dc4fd8b23f292836dde54f, State: Running, Role: LEADER
I20260620 20:22:36.714072 4370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b5153217dc4fd8b23f292836dde54f" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33455 } }
I20260620 20:22:36.714572 4365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:36.716786 4371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68b5153217dc4fd8b23f292836dde54f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b5153217dc4fd8b23f292836dde54f" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33455 } } }
I20260620 20:22:36.716825 4372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68b5153217dc4fd8b23f292836dde54f. Latest consensus state: current_term: 1 leader_uuid: "68b5153217dc4fd8b23f292836dde54f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b5153217dc4fd8b23f292836dde54f" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33455 } } }
I20260620 20:22:36.717269 4371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:36.717375 4372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:36.718881 4375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:36.723456 4375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:36.735575 4375 catalog_manager.cc:1347] Generated new cluster ID: eb459b5e09dd47198dd3428e5a352144
I20260620 20:22:36.735688 4375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:36.767112 4375 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:36.769119 4375 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:36.780743 4375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f: Generated new TSK 0
I20260620 20:22:36.781845 4375 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
I20260620 20:22:36.835220 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33455
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:37.164341 4391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:37.164629 4391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:37.164736 4391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:37.172214 4391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:37.172353 4391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:37.172482 4391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:22:37.182626 4391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:37.183914 4391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:37.185211 4391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:37.198585 4397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:37.198630 4399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:37.198654 4396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:37.199753 4391 server_base.cc:1034] running on GCE node
I20260620 20:22:37.200440 4391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:37.201535 4391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:37.202781 4391 hybrid_clock.cc:648] HybridClock initialized: now 1781986957202735 us; error 42 us; skew 500 ppm
I20260620 20:22:37.203181 4391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:37 dist-test-slave-zzwf krb5kdc[4285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986957, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:37.206355 4391 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:22:37.207428 4391 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:37.207516 4391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:37.207765 4391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:37.210731 4391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "1860015164c6479a83457eaedc77294c"
format_stamp: "Formatted at 2026-06-20 20:22:37 on dist-test-slave-zzwf"
server_key: "80b83cdac081f1eb6ea6eb7d6368deed"
server_key_iv: "cbc9b2b65e83a2b8116fc9d4421b1b8d"
server_key_version: "encryptionkey@0"
I20260620 20:22:37.211478 4391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "1860015164c6479a83457eaedc77294c"
format_stamp: "Formatted at 2026-06-20 20:22:37 on dist-test-slave-zzwf"
server_key: "80b83cdac081f1eb6ea6eb7d6368deed"
server_key_iv: "cbc9b2b65e83a2b8116fc9d4421b1b8d"
server_key_version: "encryptionkey@0"
I20260620 20:22:37.216748 4391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:22:37.220410 4405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:37.221817 4391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:37.222004 4391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "1860015164c6479a83457eaedc77294c"
format_stamp: "Formatted at 2026-06-20 20:22:37 on dist-test-slave-zzwf"
server_key: "80b83cdac081f1eb6ea6eb7d6368deed"
server_key_iv: "cbc9b2b65e83a2b8116fc9d4421b1b8d"
server_key_version: "encryptionkey@0"
I20260620 20:22:37.222232 4391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:37.244433 4391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:37.248912 4391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:37.249197 4391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:37.250226 4391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:37.251603 4391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:37.251691 4391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:37.251842 4391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:37.251941 4391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:37.283820 4391 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:35297
I20260620 20:22:37.283862 4517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:35297 every 8 connection(s)
I20260620 20:22:37.284682 4391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:22:37.294375 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 4391
I20260620 20:22:37.294621 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:22:37.295367 32686 external_mini_cluster.cc:1428] Setting key aa9216f0eaabdbc1448cc1574942f4c7
Jun 20 20:22:37 dist-test-slave-zzwf krb5kdc[4285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986957, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:37.304245 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:37.287193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:38337 (local address 127.31.235.190:33455)
0620 20:22:37.287641 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:37.287645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:37.288656 (+ 1011us) server_negotiation.cc:407] Connection header received
0620 20:22:37.290535 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:37.290541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:37.290637 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:37.290758 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:37.292216 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:37.292923 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:37.294517 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:37.295017 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:37.295146 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:37.298745 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:37.298749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:37.298752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:37.298782 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:37.301269 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:37.302070 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:37.302082 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:37.302085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:37.302189 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:37.302596 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:37.302601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:37.302605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:37.302852 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:37.303000 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:37.303675 (+ 675us) server_negotiation.cc:299] Negotiation successful
0620 20:22:37.303887 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":232,"threads_started":1}
I20260620 20:22:37.305224 4522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:37.287679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33455 (local address 127.31.235.129:38337)
0620 20:22:37.288504 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:37.288542 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:37.290268 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:37.290975 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:37.290983 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:37.291514 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:37.292055 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:37.292071 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:37.293072 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:37.293077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:37.294368 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:37.294379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:37.295276 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:37.295283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:37.295583 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:37.296372 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:37.296394 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:37.298529 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:37.301420 (+ 2891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:37.301430 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:37.301447 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:37.301940 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:37.302295 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:37.302301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:37.302305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:37.302481 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:37.302975 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:37.302982 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:37.303307 (+ 325us) client_negotiation.cc:769] Sending connection context
0620 20:22:37.303564 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 20:22:37.304093 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":321,"threads_started":1}
I20260620 20:22:37.306802 4518 heartbeater.cc:347] Connected to a master server at 127.31.235.190:33455
I20260620 20:22:37.307130 4518 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:37.307758 4518 heartbeater.cc:511] Master 127.31.235.190:33455 requested a full tablet report, sending...
I20260620 20:22:37.310104 4329 ts_manager.cc:194] Registered new tserver with Master: 1860015164c6479a83457eaedc77294c (127.31.235.129:35297)
I20260620 20:22:37.312322 4329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:38337
WARNING: no policy specified for kudu/127.31.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
I20260620 20:22:37.340580 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33455
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:37.660619 4525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:37.660905 4525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:37.661067 4525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:37.668363 4525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:37.668476 4525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:37.668653 4525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:22:37.679049 4525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:37.680390 4525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:37.681769 4525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:37.695104 4530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:37.695129 4531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:37.695358 4533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:37.696086 4525 server_base.cc:1034] running on GCE node
I20260620 20:22:37.696699 4525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:37.697705 4525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:37.698952 4525 hybrid_clock.cc:648] HybridClock initialized: now 1781986957698898 us; error 62 us; skew 500 ppm
I20260620 20:22:37.699291 4525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:37 dist-test-slave-zzwf krb5kdc[4285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986957, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:37.702561 4525 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:22:37.703823 4525 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:37.703964 4525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:37.704284 4525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:37.707270 4525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "53939fd4422847879f2a207ee12394b9"
format_stamp: "Formatted at 2026-06-20 20:22:37 on dist-test-slave-zzwf"
server_key: "f61112795921a12fa2c5296eee37144a"
server_key_iv: "11c0f71d620c2d9e11f854b3351e6f87"
server_key_version: "encryptionkey@0"
I20260620 20:22:37.708103 4525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "53939fd4422847879f2a207ee12394b9"
format_stamp: "Formatted at 2026-06-20 20:22:37 on dist-test-slave-zzwf"
server_key: "f61112795921a12fa2c5296eee37144a"
server_key_iv: "11c0f71d620c2d9e11f854b3351e6f87"
server_key_version: "encryptionkey@0"
I20260620 20:22:37.713200 4525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 20:22:37.716993 4539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:37.718400 4525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:37.718577 4525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "53939fd4422847879f2a207ee12394b9"
format_stamp: "Formatted at 2026-06-20 20:22:37 on dist-test-slave-zzwf"
server_key: "f61112795921a12fa2c5296eee37144a"
server_key_iv: "11c0f71d620c2d9e11f854b3351e6f87"
server_key_version: "encryptionkey@0"
I20260620 20:22:37.718801 4525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:37.768247 4525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:37.772549 4525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:37.772845 4525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:37.773916 4525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:37.775303 4525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:37.775453 4525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:37.775583 4525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:37.775669 4525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:37.809607 4525 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:43409
I20260620 20:22:37.809687 4652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:43409 every 8 connection(s)
I20260620 20:22:37.810519 4525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:22:37.820425 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 4525
I20260620 20:22:37.820624 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:22:37.821100 32686 external_mini_cluster.cc:1428] Setting key dc3b3853730b8b0588ef0344c41d3e60
Jun 20 20:22:37 dist-test-slave-zzwf krb5kdc[4285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986957, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:37.831269 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:37.813137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:55227 (local address 127.31.235.190:33455)
0620 20:22:37.813590 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:37.813595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:37.814697 (+ 1102us) server_negotiation.cc:407] Connection header received
0620 20:22:37.816577 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:37.816584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:37.816684 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:37.816800 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:37.818359 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:37.819053 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:37.820876 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:37.821368 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:37.821471 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:37.825409 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:37.825416 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:37.825423 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:37.825475 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:37.828173 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:37.828895 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:37.828900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:37.828904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:37.829021 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:37.829521 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:37.829525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:37.829533 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:37.829785 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:37.829938 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:37.830686 (+ 748us) server_negotiation.cc:299] Negotiation successful
0620 20:22:37.830892 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":194,"threads_started":1}
I20260620 20:22:37.832319 4656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:37.813776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33455 (local address 127.31.235.130:55227)
0620 20:22:37.814541 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:37.814584 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:37.816305 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:37.817021 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:37.817029 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:37.817600 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:37.818190 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:37.818206 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:37.819200 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:37.819210 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:37.820730 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:37.820746 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:37.821630 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:37.821637 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:37.821859 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:37.822592 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:37.822615 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:37.825160 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:37.828328 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:37.828337 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:37.828354 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:37.828768 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:37.829128 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:37.829133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:37.829136 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:37.829375 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:37.829930 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:37.829938 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:37.830277 (+ 339us) client_negotiation.cc:769] Sending connection context
0620 20:22:37.830594 (+ 317us) client_negotiation.cc:240] Negotiation successful
0620 20:22:37.831139 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":277,"threads_started":1}
I20260620 20:22:37.833884 4653 heartbeater.cc:347] Connected to a master server at 127.31.235.190:33455
I20260620 20:22:37.834232 4653 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:37.834995 4653 heartbeater.cc:511] Master 127.31.235.190:33455 requested a full tablet report, sending...
I20260620 20:22:37.836727 4329 ts_manager.cc:194] Registered new tserver with Master: 53939fd4422847879f2a207ee12394b9 (127.31.235.130:43409)
I20260620 20:22:37.838055 4329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:55227
WARNING: no policy specified for kudu/127.31.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
I20260620 20:22:37.869339 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33455
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:36043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:38.195575 4659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:38.195854 4659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:38.196027 4659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:38.203299 4659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:38.203420 4659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:38.203601 4659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:22:38.213970 4659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:36043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:38.215385 4659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:38.216859 4659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:38.230635 4664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:38.230613 4665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:38.230662 4667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:38.231518 4659 server_base.cc:1034] running on GCE node
I20260620 20:22:38.232141 4659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:38.233163 4659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:38.234408 4659 hybrid_clock.cc:648] HybridClock initialized: now 1781986958234311 us; error 81 us; skew 500 ppm
I20260620 20:22:38.234748 4659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:38 dist-test-slave-zzwf krb5kdc[4285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986958, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:38.238137 4659 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:22:38.239391 4659 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:38.239529 4659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:38.239856 4659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:38.242969 4659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "b3a58f4fd3df4d098fbe332dcd66ba2c"
format_stamp: "Formatted at 2026-06-20 20:22:38 on dist-test-slave-zzwf"
server_key: "a6e635711116b71cbca1a224dbfc460e"
server_key_iv: "86c4355bb38395aab43c21c60a22011e"
server_key_version: "encryptionkey@0"
I20260620 20:22:38.243783 4659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "b3a58f4fd3df4d098fbe332dcd66ba2c"
format_stamp: "Formatted at 2026-06-20 20:22:38 on dist-test-slave-zzwf"
server_key: "a6e635711116b71cbca1a224dbfc460e"
server_key_iv: "86c4355bb38395aab43c21c60a22011e"
server_key_version: "encryptionkey@0"
I20260620 20:22:38.249459 4659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 20:22:38.254580 4673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:38.256498 4659 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260620 20:22:38.256699 4659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "b3a58f4fd3df4d098fbe332dcd66ba2c"
format_stamp: "Formatted at 2026-06-20 20:22:38 on dist-test-slave-zzwf"
server_key: "a6e635711116b71cbca1a224dbfc460e"
server_key_iv: "86c4355bb38395aab43c21c60a22011e"
server_key_version: "encryptionkey@0"
I20260620 20:22:38.256951 4659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:38.283638 4659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:38.289821 4659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:38.290133 4659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:38.291241 4659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:38.292575 4659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:38.292660 4659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:38.292732 4659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:38.292788 4659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:38.315796 4518 heartbeater.cc:503] Master 127.31.235.190:33455 was elected leader, sending a full tablet report...
I20260620 20:22:38.326280 4659 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:46011
I20260620 20:22:38.326380 4785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:46011 every 8 connection(s)
I20260620 20:22:38.327281 4659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:22:38.329900 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 4659
I20260620 20:22:38.330101 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:22:38.330622 32686 external_mini_cluster.cc:1428] Setting key 8ccc1f5b3b3c9d36968b880ef1d66c24
Jun 20 20:22:38 dist-test-slave-zzwf krb5kdc[4285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986958, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:38.347564 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:38.329716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:41537 (local address 127.31.235.190:33455)
0620 20:22:38.329936 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:38.329943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:38.331334 (+ 1391us) server_negotiation.cc:407] Connection header received
0620 20:22:38.333740 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:38.333748 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:38.333848 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:38.333943 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:38.335358 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.336098 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.337795 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.338313 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.338424 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:38.341746 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:38.341751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:38.341754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:38.341784 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:38.344313 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:38.345249 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:38.345252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:38.345255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:38.345384 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:38.345828 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:38.345833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:38.345836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:38.346186 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:38.346350 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:38.347017 (+ 667us) server_negotiation.cc:299] Negotiation successful
0620 20:22:38.347227 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:22:38.348483 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:38.330286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33455 (local address 127.31.235.131:41537)
0620 20:22:38.331176 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:38.331224 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:38.333407 (+ 2183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:38.334140 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:38.334148 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:38.334626 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:38.335171 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:38.335187 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.336268 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.336272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:38.337674 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:38.337686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.338516 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.338521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:38.338710 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:38.339437 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:38.339459 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:38.341485 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:38.344460 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:38.344471 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:38.344491 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:38.345110 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:38.345515 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:38.345522 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:38.345526 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:38.345708 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:38.346315 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:38.346323 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:38.346655 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 20:22:38.346914 (+ 259us) client_negotiation.cc:240] Negotiation successful
0620 20:22:38.347424 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":361,"threads_started":1}
I20260620 20:22:38.349824 4786 heartbeater.cc:347] Connected to a master server at 127.31.235.190:33455
I20260620 20:22:38.350136 4786 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:38.350791 4786 heartbeater.cc:511] Master 127.31.235.190:33455 requested a full tablet report, sending...
I20260620 20:22:38.352285 4329 ts_manager.cc:194] Registered new tserver with Master: b3a58f4fd3df4d098fbe332dcd66ba2c (127.31.235.131:46011)
I20260620 20:22:38.353098 4329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:41537
I20260620 20:22:38.358381 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:22:38.375644 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:38.363097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34704 (local address 127.31.235.190:33455)
0620 20:22:38.363268 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:38.363273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:38.363683 (+ 410us) server_negotiation.cc:407] Connection header received
0620 20:22:38.363811 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:38.363814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:38.363886 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:38.363974 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:38.365385 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.366170 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.367399 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.367839 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.367949 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:38.370297 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:38.370302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:38.370305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:38.370353 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:38.372967 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:38.373738 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:38.373742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:38.373745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:38.373865 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:38.374395 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:38.374399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:38.374401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:38.374729 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:38.374922 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:38.375228 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 20:22:38.375381 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:22:38.379755 4329 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:22:38.386659 4329 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:22:38.425275 4798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:38.399098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:43409 (local address 127.0.0.1:53212)
0620 20:22:38.414171 (+ 15073us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:38.414193 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:38.414346 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:38.417944 (+ 3598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:38.417949 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:38.417981 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:38.418383 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:38.418394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.420164 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.420169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:38.422208 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:38.422220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.424206 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.424212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:38.424397 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:38.424462 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 20:22:38.424574 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 20:22:38.424771 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14942,"mutex_wait_us":23,"thread_start_us":13346,"threads_started":1}
I20260620 20:22:38.425793 4799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:38.399004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:35297 (local address 127.0.0.1:37184)
0620 20:22:38.411807 (+ 12803us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:38.411880 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:38.412168 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:38.418549 (+ 6381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:38.418554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:38.418580 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:38.418877 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:38.418887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.420949 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.420954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:38.423199 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:38.423211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.424969 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.424977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:38.425141 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:38.425182 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 20:22:38.425280 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 20:22:38.425406 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12401,"thread_start_us":15449,"threads_started":1}
I20260620 20:22:38.425817 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:38.413899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45216 (local address 127.31.235.131:46011)
0620 20:22:38.416047 (+ 2148us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:38.416058 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:38.416280 (+ 222us) server_negotiation.cc:407] Connection header received
0620 20:22:38.416381 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:38.416393 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:38.416679 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:38.417630 (+ 951us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:38.417947 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.419776 (+ 1829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.422578 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.424120 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.424312 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:38.424489 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:38.425232 (+ 743us) server_negotiation.cc:299] Negotiation successful
0620 20:22:38.425471 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1968,"thread_start_us":1565,"threads_started":1}
I20260620 20:22:38.425464 4803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:38.414861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:46011 (local address 127.0.0.1:45216)
0620 20:22:38.416036 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:38.416052 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:38.416167 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:38.416917 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:38.416925 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:38.416967 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:38.417343 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:38.417356 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.419918 (+ 2562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.419923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:38.422440 (+ 2517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:38.422454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.424625 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.424630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:38.424783 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:38.424812 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 20:22:38.424899 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 20:22:38.425031 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"mutex_wait_us":87,"thread_start_us":1201,"threads_started":1}
I20260620 20:22:38.426062 4800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:38.399167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37184 (local address 127.31.235.129:35297)
0620 20:22:38.417962 (+ 18795us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:38.417971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:38.417997 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:22:38.418097 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:38.418107 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:38.418329 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:38.418532 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:38.419062 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.420813 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.423370 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.424758 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.424964 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:38.425124 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:38.425597 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 20:22:38.425748 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18608,"thread_start_us":18465,"threads_started":1}
I20260620 20:22:38.427881 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:38.399478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53212 (local address 127.31.235.130:43409)
0620 20:22:38.417274 (+ 17796us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:38.417285 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:38.417317 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:22:38.417411 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:38.417422 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:38.417681 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:38.417932 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:38.418572 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.420010 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.422366 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.423967 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.427151 (+ 3184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:38.427322 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:38.427405 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 20:22:38.427564 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17623,"thread_start_us":17684,"threads_started":1}
I20260620 20:22:38.429946 4721 tablet_service.cc:1467] Processing CreateTablet for tablet bce4ff4aef62495cb6507fa175271c62 (DEFAULT_TABLE table=test-table [id=50891f18339848679e91f6b01a223bcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:38.429800 4453 tablet_service.cc:1467] Processing CreateTablet for tablet bce4ff4aef62495cb6507fa175271c62 (DEFAULT_TABLE table=test-table [id=50891f18339848679e91f6b01a223bcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:38.431195 4721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bce4ff4aef62495cb6507fa175271c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:38.431222 4453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bce4ff4aef62495cb6507fa175271c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:38.431394 4587 tablet_service.cc:1467] Processing CreateTablet for tablet bce4ff4aef62495cb6507fa175271c62 (DEFAULT_TABLE table=test-table [id=50891f18339848679e91f6b01a223bcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:38.432578 4587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bce4ff4aef62495cb6507fa175271c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:38.442878 4804 tablet_bootstrap.cc:492] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c: Bootstrap starting.
I20260620 20:22:38.446111 4804 tablet_bootstrap.cc:654] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:38.446189 4805 tablet_bootstrap.cc:492] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c: Bootstrap starting.
I20260620 20:22:38.447777 4806 tablet_bootstrap.cc:492] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9: Bootstrap starting.
I20260620 20:22:38.448026 4804 log.cc:824] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:38.450368 4805 tablet_bootstrap.cc:654] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:38.450853 4806 tablet_bootstrap.cc:654] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:38.451386 4804 tablet_bootstrap.cc:492] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c: No bootstrap required, opened a new log
I20260620 20:22:38.451622 4805 log.cc:824] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:38.451668 4804 ts_tablet_manager.cc:1397] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 20:22:38.451850 4806 log.cc:824] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:38.454558 4805 tablet_bootstrap.cc:492] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c: No bootstrap required, opened a new log
I20260620 20:22:38.454550 4806 tablet_bootstrap.cc:492] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9: No bootstrap required, opened a new log
I20260620 20:22:38.454808 4805 ts_tablet_manager.cc:1397] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 20:22:38.454813 4806 ts_tablet_manager.cc:1397] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.005s
I20260620 20:22:38.459941 4804 raft_consensus.cc:348] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1860015164c6479a83457eaedc77294c" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35297 } } peers { permanent_uuid: "53939fd4422847879f2a207ee12394b9" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43409 } } peers { permanent_uuid: "b3a58f4fd3df4d098fbe332dcd66ba2c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46011 } }
I20260620 20:22:38.460420 4804 raft_consensus.cc:374] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:38.460522 4804 raft_consensus.cc:729] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3a58f4fd3df4d098fbe332dcd66ba2c, State: Initialized, Role: FOLLOWER
I20260620 20:22:38.461184 4804 consensus_queue.cc:260] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1860015164c6479a83457eaedc77294c" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35297 } } peers { permanent_uuid: "53939fd4422847879f2a207ee12394b9" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43409 } } peers { permanent_uuid: "b3a58f4fd3df4d098fbe332dcd66ba2c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46011 } }
I20260620 20:22:38.465453 4805 raft_consensus.cc:348] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1860015164c6479a83457eaedc77294c" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35297 } } peers { permanent_uuid: "53939fd4422847879f2a207ee12394b9" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43409 } } peers { permanent_uuid: "b3a58f4fd3df4d098fbe332dcd66ba2c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46011 } }
I20260620 20:22:38.465452 4806 raft_consensus.cc:348] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1860015164c6479a83457eaedc77294c" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35297 } } peers { permanent_uuid: "53939fd4422847879f2a207ee12394b9" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43409 } } peers { permanent_uuid: "b3a58f4fd3df4d098fbe332dcd66ba2c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46011 } }
I20260620 20:22:38.465904 4805 raft_consensus.cc:374] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:38.465904 4806 raft_consensus.cc:374] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:38.466019 4806 raft_consensus.cc:729] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53939fd4422847879f2a207ee12394b9, State: Initialized, Role: FOLLOWER
I20260620 20:22:38.466007 4805 raft_consensus.cc:729] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1860015164c6479a83457eaedc77294c, State: Initialized, Role: FOLLOWER
I20260620 20:22:38.466429 4786 heartbeater.cc:503] Master 127.31.235.190:33455 was elected leader, sending a full tablet report...
I20260620 20:22:38.466643 4805 consensus_queue.cc:260] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1860015164c6479a83457eaedc77294c" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35297 } } peers { permanent_uuid: "53939fd4422847879f2a207ee12394b9" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43409 } } peers { permanent_uuid: "b3a58f4fd3df4d098fbe332dcd66ba2c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46011 } }
I20260620 20:22:38.466673 4806 consensus_queue.cc:260] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1860015164c6479a83457eaedc77294c" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35297 } } peers { permanent_uuid: "53939fd4422847879f2a207ee12394b9" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43409 } } peers { permanent_uuid: "b3a58f4fd3df4d098fbe332dcd66ba2c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46011 } }
I20260620 20:22:38.466982 4804 ts_tablet_manager.cc:1428] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c: Time spent starting tablet: real 0.015s user 0.009s sys 0.007s
I20260620 20:22:38.470063 4653 heartbeater.cc:503] Master 127.31.235.190:33455 was elected leader, sending a full tablet report...
I20260620 20:22:38.470222 4805 ts_tablet_manager.cc:1428] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260620 20:22:38.470531 4806 ts_tablet_manager.cc:1428] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
W20260620 20:22:38.537077 4520 tablet.cc:2369] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:22:38.562459 4654 tablet.cc:2369] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:22:38.578774 4787 tablet.cc:2369] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:22:38.635550 4811 raft_consensus.cc:482] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:22:38.635866 4811 raft_consensus.cc:504] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1860015164c6479a83457eaedc77294c" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35297 } } peers { permanent_uuid: "53939fd4422847879f2a207ee12394b9" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43409 } } peers { permanent_uuid: "b3a58f4fd3df4d098fbe332dcd66ba2c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46011 } }
I20260620 20:22:38.637655 4811 leader_election.cc:288] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53939fd4422847879f2a207ee12394b9 (127.31.235.130:43409), b3a58f4fd3df4d098fbe332dcd66ba2c (127.31.235.131:46011)
I20260620 20:22:38.644712 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:38.638090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:41139 (local address 127.31.235.130:43409)
0620 20:22:38.638304 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:38.638313 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:38.638774 (+ 461us) server_negotiation.cc:407] Connection header received
0620 20:22:38.639056 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:38.639063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:38.639154 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:38.639223 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:38.640017 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.640845 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.642865 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.643696 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.643824 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:38.643875 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:38.644288 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 20:22:38.644442 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:22:38.644797 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:38.638938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:52113 (local address 127.31.235.131:46011)
0620 20:22:38.639129 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:38.639136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:38.639479 (+ 343us) server_negotiation.cc:407] Connection header received
0620 20:22:38.639631 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:38.639638 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:38.639714 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:38.639818 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:38.640356 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.641130 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.643041 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.643857 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.644016 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:38.644085 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:38.644428 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 20:22:38.644571 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:22:38.644905 4813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:38.638047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:43409 (local address 127.31.235.129:41139)
0620 20:22:38.638650 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:38.638669 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:38.638876 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:38.639391 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:38.639394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:38.639423 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:38.639854 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:38.639867 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.640971 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.640976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:38.642664 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:38.642676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.643925 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.643933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:38.644139 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:38.644162 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:22:38.644234 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 20:22:38.644387 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":44,"thread_start_us":326,"threads_started":1}
I20260620 20:22:38.644951 4814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:38.638802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:46011 (local address 127.31.235.129:52113)
0620 20:22:38.639371 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:38.639384 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:38.639501 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:38.639835 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:38.639838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:38.639854 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:38.640204 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:38.640215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.641333 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.641337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:38.642887 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:38.642900 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.644053 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.644057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:38.644181 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:38.644203 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:22:38.644392 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 20:22:38.644512 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":1,"thread_start_us":293,"threads_started":1}
I20260620 20:22:38.645216 4812 raft_consensus.cc:482] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:22:38.645548 4812 raft_consensus.cc:504] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1860015164c6479a83457eaedc77294c" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35297 } } peers { permanent_uuid: "53939fd4422847879f2a207ee12394b9" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43409 } } peers { permanent_uuid: "b3a58f4fd3df4d098fbe332dcd66ba2c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46011 } }
I20260620 20:22:38.646023 4741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce4ff4aef62495cb6507fa175271c62" candidate_uuid: "1860015164c6479a83457eaedc77294c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3a58f4fd3df4d098fbe332dcd66ba2c" is_pre_election: true
I20260620 20:22:38.646549 4741 raft_consensus.cc:2454] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1860015164c6479a83457eaedc77294c in term 0.
I20260620 20:22:38.646548 4607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce4ff4aef62495cb6507fa175271c62" candidate_uuid: "1860015164c6479a83457eaedc77294c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53939fd4422847879f2a207ee12394b9" is_pre_election: true
I20260620 20:22:38.646983 4607 raft_consensus.cc:2454] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1860015164c6479a83457eaedc77294c in term 0.
I20260620 20:22:38.647244 4406 leader_election.cc:302] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1860015164c6479a83457eaedc77294c, b3a58f4fd3df4d098fbe332dcd66ba2c; no voters:
I20260620 20:22:38.647560 4812 leader_election.cc:288] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1860015164c6479a83457eaedc77294c (127.31.235.129:35297), b3a58f4fd3df4d098fbe332dcd66ba2c (127.31.235.131:46011)
I20260620 20:22:38.647671 4811 raft_consensus.cc:2781] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:22:38.647799 4811 raft_consensus.cc:482] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:22:38.647898 4811 raft_consensus.cc:3037] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:38.651196 4811 raft_consensus.cc:504] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1860015164c6479a83457eaedc77294c" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35297 } } peers { permanent_uuid: "53939fd4422847879f2a207ee12394b9" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43409 } } peers { permanent_uuid: "b3a58f4fd3df4d098fbe332dcd66ba2c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46011 } }
I20260620 20:22:38.651870 4811 leader_election.cc:288] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c [CANDIDATE]: Term 1 election: Requested vote from peers 53939fd4422847879f2a207ee12394b9 (127.31.235.130:43409), b3a58f4fd3df4d098fbe332dcd66ba2c (127.31.235.131:46011)
I20260620 20:22:38.652352 4607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce4ff4aef62495cb6507fa175271c62" candidate_uuid: "1860015164c6479a83457eaedc77294c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53939fd4422847879f2a207ee12394b9"
I20260620 20:22:38.652595 4607 raft_consensus.cc:3037] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:38.652611 4741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce4ff4aef62495cb6507fa175271c62" candidate_uuid: "1860015164c6479a83457eaedc77294c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3a58f4fd3df4d098fbe332dcd66ba2c"
I20260620 20:22:38.652818 4741 raft_consensus.cc:3037] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:38.656033 4607 raft_consensus.cc:2454] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1860015164c6479a83457eaedc77294c in term 1.
I20260620 20:22:38.656263 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:38.648208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:47655 (local address 127.31.235.131:46011)
0620 20:22:38.648897 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:38.648903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:38.648921 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:22:38.648977 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:38.648982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:38.649047 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:38.649130 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:38.649899 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.650790 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.653308 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.654690 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.654880 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:38.654953 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:38.655809 (+ 856us) server_negotiation.cc:299] Negotiation successful
0620 20:22:38.655969 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572}
I20260620 20:22:38.656222 4741 raft_consensus.cc:2454] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1860015164c6479a83457eaedc77294c in term 1.
I20260620 20:22:38.656481 4406 leader_election.cc:302] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1860015164c6479a83457eaedc77294c, 53939fd4422847879f2a207ee12394b9; no voters:
I20260620 20:22:38.656792 4811 raft_consensus.cc:2781] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:38.657219 4815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:38.647831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:46011 (local address 127.31.235.130:47655)
0620 20:22:38.648538 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:38.648556 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:38.648672 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:38.649181 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:38.649189 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:38.649231 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:38.649754 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:38.649773 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.650972 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.650978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:38.653145 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:38.653160 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.655387 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.655393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:38.655619 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:38.655652 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 20:22:38.656621 (+ 969us) client_negotiation.cc:240] Negotiation successful
0620 20:22:38.656764 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":24,"thread_start_us":424,"threads_started":1}
I20260620 20:22:38.657950 4741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce4ff4aef62495cb6507fa175271c62" candidate_uuid: "53939fd4422847879f2a207ee12394b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3a58f4fd3df4d098fbe332dcd66ba2c" is_pre_election: true
I20260620 20:22:38.658258 4741 raft_consensus.cc:2379] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 53939fd4422847879f2a207ee12394b9 in current term 1: Already voted for candidate 1860015164c6479a83457eaedc77294c in this term.
I20260620 20:22:38.658336 4800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:38.648742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:54577 (local address 127.31.235.129:35297)
0620 20:22:38.648992 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:38.649004 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:38.649922 (+ 918us) server_negotiation.cc:407] Connection header received
0620 20:22:38.650149 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:38.650154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:38.650229 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:38.650327 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:38.650879 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.652087 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.655757 (+ 3670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.657063 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.657224 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:38.657321 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:38.657954 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 20:22:38.658096 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:22:38.659142 4816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:38.648122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:35297 (local address 127.31.235.130:54577)
0620 20:22:38.649825 (+ 1703us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:38.649850 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:38.650018 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:38.650371 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:38.650376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:38.650400 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:38.650736 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:38.650746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.653359 (+ 2613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.653364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:38.655588 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:38.655602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.657579 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.657587 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:38.657754 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:38.657788 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 20:22:38.658484 (+ 696us) client_negotiation.cc:240] Negotiation successful
0620 20:22:38.658612 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1583,"mutex_wait_us":2,"thread_start_us":1366,"threads_started":1}
I20260620 20:22:38.659760 4811 raft_consensus.cc:686] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c [term 1 LEADER]: Becoming Leader. State: Replica: 1860015164c6479a83457eaedc77294c, State: Running, Role: LEADER
I20260620 20:22:38.660183 4473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bce4ff4aef62495cb6507fa175271c62" candidate_uuid: "53939fd4422847879f2a207ee12394b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1860015164c6479a83457eaedc77294c" is_pre_election: true
I20260620 20:22:38.660321 4811 consensus_queue.cc:237] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1860015164c6479a83457eaedc77294c" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35297 } } peers { permanent_uuid: "53939fd4422847879f2a207ee12394b9" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43409 } } peers { permanent_uuid: "b3a58f4fd3df4d098fbe332dcd66ba2c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46011 } }
I20260620 20:22:38.661311 4542 leader_election.cc:302] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53939fd4422847879f2a207ee12394b9; no voters: 1860015164c6479a83457eaedc77294c, b3a58f4fd3df4d098fbe332dcd66ba2c
I20260620 20:22:38.661746 4812 raft_consensus.cc:2726] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 20:22:38.665642 4327 catalog_manager.cc:5495] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1860015164c6479a83457eaedc77294c (127.31.235.129). New cstate: current_term: 1 leader_uuid: "1860015164c6479a83457eaedc77294c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1860015164c6479a83457eaedc77294c" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53939fd4422847879f2a207ee12394b9" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3a58f4fd3df4d098fbe332dcd66ba2c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46011 } health_report { overall_health: UNKNOWN } } }
I20260620 20:22:38.727495 4800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:38.721876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37196 (local address 127.31.235.129:35297)
0620 20:22:38.722110 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:38.722117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:38.722258 (+ 141us) server_negotiation.cc:407] Connection header received
0620 20:22:38.722511 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:38.722515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:38.722590 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:38.722740 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:38.723104 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.723848 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.725163 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.725653 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.725783 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:38.726258 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:38.726842 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:38.727006 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:38.727083 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 20:22:38.727160 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:22:38.735546 4741 raft_consensus.cc:1261] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c [term 1 FOLLOWER]: Refusing update from remote peer 1860015164c6479a83457eaedc77294c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:38.735570 4607 raft_consensus.cc:1261] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9 [term 1 FOLLOWER]: Refusing update from remote peer 1860015164c6479a83457eaedc77294c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:38.736430 4817 consensus_queue.cc:1035] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3a58f4fd3df4d098fbe332dcd66ba2c" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:38.736778 4811 consensus_queue.cc:1035] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c [LEADER]: Connected to new peer: Peer: permanent_uuid: "53939fd4422847879f2a207ee12394b9" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:38.764343 4327 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:38.764657 4327 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:38.769660 4327 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68b5153217dc4fd8b23f292836dde54f: Sending DeleteTablet for 3 replicas of tablet bce4ff4aef62495cb6507fa175271c62
I20260620 20:22:38.770911 4453 tablet_service.cc:1514] Processing DeleteTablet for tablet bce4ff4aef62495cb6507fa175271c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:38 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:37184
I20260620 20:22:38.771014 4587 tablet_service.cc:1514] Processing DeleteTablet for tablet bce4ff4aef62495cb6507fa175271c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:38 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:53212
I20260620 20:22:38.771250 4721 tablet_service.cc:1514] Processing DeleteTablet for tablet bce4ff4aef62495cb6507fa175271c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:38 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:45216
I20260620 20:22:38.774046 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 4391
I20260620 20:22:38.780342 4832 tablet_replica.cc:330] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c: stopping tablet replica
I20260620 20:22:38.781719 4832 raft_consensus.cc:2229] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:22:38.783193 4832 raft_consensus.cc:2258] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:38.786988 4832 ts_tablet_manager.cc:1905] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:38.794941 4831 tablet_replica.cc:330] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9: stopping tablet replica
I20260620 20:22:38.796806 4832 ts_tablet_manager.cc:1918] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:22:38.797394 4832 log.cc:1196] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/bce4ff4aef62495cb6507fa175271c62
I20260620 20:22:38.797451 4831 raft_consensus.cc:2229] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:22:38.798067 4832 ts_tablet_manager.cc:1939] T bce4ff4aef62495cb6507fa175271c62 P b3a58f4fd3df4d098fbe332dcd66ba2c: Deleting consensus metadata
I20260620 20:22:38.800212 4313 catalog_manager.cc:4841] TS b3a58f4fd3df4d098fbe332dcd66ba2c (127.31.235.131:46011): tablet bce4ff4aef62495cb6507fa175271c62 (table test-table [id=50891f18339848679e91f6b01a223bcc]) successfully deleted
I20260620 20:22:38.800529 4831 raft_consensus.cc:2258] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 20:22:38 dist-test-slave-zzwf krb5kdc[4285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:22:38.803655 4831 ts_tablet_manager.cc:1905] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:38.804375 4830 tablet_replica.cc:330] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c: stopping tablet replica
I20260620 20:22:38.804869 4830 raft_consensus.cc:2229] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:22:38.805696 4830 raft_consensus.cc:2258] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:38.807855 4830 ts_tablet_manager.cc:1905] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:38.812835 4831 ts_tablet_manager.cc:1918] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:22:38.813007 4831 log.cc:1196] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/bce4ff4aef62495cb6507fa175271c62
I20260620 20:22:38.813597 4831 ts_tablet_manager.cc:1939] T bce4ff4aef62495cb6507fa175271c62 P 53939fd4422847879f2a207ee12394b9: Deleting consensus metadata
I20260620 20:22:38.815471 4313 catalog_manager.cc:4841] TS 53939fd4422847879f2a207ee12394b9 (127.31.235.130:43409): tablet bce4ff4aef62495cb6507fa175271c62 (table test-table [id=50891f18339848679e91f6b01a223bcc]) successfully deleted
I20260620 20:22:38.815811 4830 ts_tablet_manager.cc:1918] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:22:38.816001 4830 log.cc:1196] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/bce4ff4aef62495cb6507fa175271c62
I20260620 20:22:38.816553 4830 ts_tablet_manager.cc:1939] T bce4ff4aef62495cb6507fa175271c62 P 1860015164c6479a83457eaedc77294c: Deleting consensus metadata
I20260620 20:22:38.818264 4315 catalog_manager.cc:4841] TS 1860015164c6479a83457eaedc77294c (127.31.235.129:35297): tablet bce4ff4aef62495cb6507fa175271c62 (table test-table [id=50891f18339848679e91f6b01a223bcc]) successfully deleted
I20260620 20:22:38.821184 4800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:38.775482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37208 (local address 127.31.235.129:35297)
0620 20:22:38.775765 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:38.775775 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:38.787799 (+ 12024us) server_negotiation.cc:407] Connection header received
0620 20:22:38.789031 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:38.789039 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:38.789127 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:38.789220 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:38.791117 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.792415 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.795184 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.795984 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.796092 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:38.803723 (+ 7631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:38.803739 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:38.803760 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:38.803814 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:38.808423 (+ 4609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:38.818212 (+ 9789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:38.818221 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:38.818225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:38.818413 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:38.818898 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:38.818903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:38.818907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:38.819344 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:38.819659 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:38.820583 (+ 924us) server_negotiation.cc:299] Negotiation successful
0620 20:22:38.820831 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 20:22:38.822568 4513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:38.927842 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 4391
I20260620 20:22:38.946125 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 4525
Jun 20 20:22:38 dist-test-slave-zzwf krb5kdc[4285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.130@KRBTEST.COM
I20260620 20:22:38.961301 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:38.946824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53226 (local address 127.31.235.130:43409)
0620 20:22:38.947016 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:38.947021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:38.947073 (+ 52us) server_negotiation.cc:407] Connection header received
0620 20:22:38.947208 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:38.947212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:38.947274 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:38.947364 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:38.948554 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.949315 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.950681 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:38.951087 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:38.951190 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:38.955387 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:38.955401 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:38.955419 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:38.955457 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:38.958628 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:38.959301 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:38.959308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:38.959311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:38.959461 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:38.959803 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:38.959807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:38.959810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:38.960188 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:38.960417 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:38.960820 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 20:22:38.961045 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 20:22:38.962105 4648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:39.053644 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 4525
I20260620 20:22:39.072433 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 4659
Jun 20 20:22:39 dist-test-slave-zzwf krb5kdc[4285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.131@KRBTEST.COM
I20260620 20:22:39.087997 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:39.073228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45228 (local address 127.31.235.131:46011)
0620 20:22:39.073430 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:39.073438 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:39.073469 (+ 31us) server_negotiation.cc:407] Connection header received
0620 20:22:39.073730 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:39.073735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:39.073812 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:39.074099 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:39.075402 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:39.076068 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:39.077360 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:39.077856 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:39.077994 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:39.082425 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:39.082439 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:39.082455 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:39.082488 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:39.085220 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:39.086025 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:39.086032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:39.086035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:39.086153 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:39.086527 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:39.086538 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:39.086541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:39.086870 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:39.087194 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:39.087482 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 20:22:39.087691 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:22:39.088651 4781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:39.177702 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 4659
I20260620 20:22:39.194377 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 4298
I20260620 20:22:39.195420 4360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:39.305545 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 4298
2026-06-20T20:22:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4357 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T20:22:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:39Z Disabled control of system clock
I20260620 20:22:39.373054 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:39153
--webserver_interface=127.31.235.190
--webserver_port=0
--builtin_ntp_servers=127.31.235.148:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:39153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 20:22:39.699648 4841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:39.699954 4841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:39.700114 4841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:39.707598 4841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:39.707723 4841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:39.707763 4841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:39.707866 4841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:39.707960 4841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:39.717621 4841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:39153
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:39153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:39.718982 4841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:39.720836 4841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:39.733210 4846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:39.733174 4849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:39.733276 4847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:39.734756 4841 server_base.cc:1034] running on GCE node
I20260620 20:22:39.735556 4841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:39.736964 4841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:39.738229 4841 hybrid_clock.cc:648] HybridClock initialized: now 1781986959738196 us; error 49 us; skew 500 ppm
I20260620 20:22:39.738606 4841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:22:39.744807 4841 webserver.cc:460] Webserver started at http://127.31.235.190:39595/ using document root <none> and password file <none>
I20260620 20:22:39.745620 4841 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:39.745748 4841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:39.746073 4841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:39.748944 4841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "09de2f15bb1247beae190ddbf0a0fda5"
format_stamp: "Formatted at 2026-06-20 20:22:39 on dist-test-slave-zzwf"
server_key: "894c10be1402e6522b84f44efdb95886"
server_key_iv: "dbe1df26b24d5d8cbb753985d47cd188"
server_key_version: "encryptionkey@0"
I20260620 20:22:39.750178 4841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "09de2f15bb1247beae190ddbf0a0fda5"
format_stamp: "Formatted at 2026-06-20 20:22:39 on dist-test-slave-zzwf"
server_key: "894c10be1402e6522b84f44efdb95886"
server_key_iv: "dbe1df26b24d5d8cbb753985d47cd188"
server_key_version: "encryptionkey@0"
I20260620 20:22:39.755638 4841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 20:22:39.759629 4855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:39.761111 4841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:22:39.761307 4841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "09de2f15bb1247beae190ddbf0a0fda5"
format_stamp: "Formatted at 2026-06-20 20:22:39 on dist-test-slave-zzwf"
server_key: "894c10be1402e6522b84f44efdb95886"
server_key_iv: "dbe1df26b24d5d8cbb753985d47cd188"
server_key_version: "encryptionkey@0"
I20260620 20:22:39.761595 4841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:39.767410 4841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:39.768206 4841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:39.768512 4841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:39.788496 4841 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:39153
I20260620 20:22:39.788547 4906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:39153 every 8 connection(s)
I20260620 20:22:39.790002 4841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:39.792883 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 4841
I20260620 20:22:39.793084 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:22:39.793987 32686 external_mini_cluster.cc:1428] Setting key a3663a943e28cc7801aede64d79372ac
I20260620 20:22:39.794819 4907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:39.803438 4910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:39.796366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33024 (local address 127.31.235.190:39153)
0620 20:22:39.797451 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:39.797467 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:39.797570 (+ 103us) server_negotiation.cc:407] Connection header received
0620 20:22:39.799677 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:39.799700 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:39.800151 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:39.800528 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:39.801710 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:39.801717 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:39.801730 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:39.801744 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:39.801817 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:39.801837 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:39.802022 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:39.802135 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 20:22:39.802463 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":412,"threads_started":1}
I20260620 20:22:39.808641 4907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5: Bootstrap starting.
I20260620 20:22:39.811882 4907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:39.812952 4907 log.cc:824] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:39.815959 4907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5: No bootstrap required, opened a new log
I20260620 20:22:39.823421 4907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09de2f15bb1247beae190ddbf0a0fda5" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39153 } }
I20260620 20:22:39.823772 4907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:39.823953 4907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09de2f15bb1247beae190ddbf0a0fda5, State: Initialized, Role: FOLLOWER
I20260620 20:22:39.824685 4907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09de2f15bb1247beae190ddbf0a0fda5" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39153 } }
I20260620 20:22:39.825012 4907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:39.825167 4907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:39.825351 4907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:39.827605 4907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09de2f15bb1247beae190ddbf0a0fda5" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39153 } }
I20260620 20:22:39.828137 4907 leader_election.cc:302] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09de2f15bb1247beae190ddbf0a0fda5; no voters:
I20260620 20:22:39.828734 4907 leader_election.cc:288] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:39.828895 4912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:39.829634 4912 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5 [term 1 LEADER]: Becoming Leader. State: Replica: 09de2f15bb1247beae190ddbf0a0fda5, State: Running, Role: LEADER
I20260620 20:22:39.830184 4912 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09de2f15bb1247beae190ddbf0a0fda5" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39153 } }
I20260620 20:22:39.831090 4907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:39.833015 4913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09de2f15bb1247beae190ddbf0a0fda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09de2f15bb1247beae190ddbf0a0fda5" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39153 } } }
I20260620 20:22:39.833379 4913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:39.833139 4914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09de2f15bb1247beae190ddbf0a0fda5. Latest consensus state: current_term: 1 leader_uuid: "09de2f15bb1247beae190ddbf0a0fda5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09de2f15bb1247beae190ddbf0a0fda5" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39153 } } }
I20260620 20:22:39.833613 4914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:39.835255 4917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:39.840338 4917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:39.850430 4917 catalog_manager.cc:1347] Generated new cluster ID: a94f28dce6c14ea1a44e1b3384c4384a
I20260620 20:22:39.850564 4917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:39.862000 4917 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:39.862454 4917 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:39.872229 4917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5: Generated new TSK 0
I20260620 20:22:39.873301 4917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:39.887303 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:39153
--builtin_ntp_servers=127.31.235.148:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 20:22:40.249246 4931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:40.249749 4931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:40.249933 4931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:40.259644 4931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:40.259750 4931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:40.259920 4931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:22:40.269307 4931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:39153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:40.270655 4931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:40.272377 4931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:40.285408 4936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:40.285458 4939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:40.286077 4937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:40.286165 4931 server_base.cc:1034] running on GCE node
I20260620 20:22:40.286725 4931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:40.287618 4931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:40.288810 4931 hybrid_clock.cc:648] HybridClock initialized: now 1781986960288774 us; error 29 us; skew 500 ppm
I20260620 20:22:40.289157 4931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:22:40.291379 4931 webserver.cc:460] Webserver started at http://127.31.235.129:33173/ using document root <none> and password file <none>
I20260620 20:22:40.292075 4931 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:40.292182 4931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:40.292503 4931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:40.295289 4931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "e1ea2eea52a94b5faad67c2a4e3a653d"
format_stamp: "Formatted at 2026-06-20 20:22:40 on dist-test-slave-zzwf"
server_key: "a65ebd73c827068d125baff62a867ed2"
server_key_iv: "8d385607af717dbcc301594fc53e3edc"
server_key_version: "encryptionkey@0"
I20260620 20:22:40.296049 4931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "e1ea2eea52a94b5faad67c2a4e3a653d"
format_stamp: "Formatted at 2026-06-20 20:22:40 on dist-test-slave-zzwf"
server_key: "a65ebd73c827068d125baff62a867ed2"
server_key_iv: "8d385607af717dbcc301594fc53e3edc"
server_key_version: "encryptionkey@0"
I20260620 20:22:40.300881 4931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 20:22:40.304306 4945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:40.305739 4931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:40.305915 4931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "e1ea2eea52a94b5faad67c2a4e3a653d"
format_stamp: "Formatted at 2026-06-20 20:22:40 on dist-test-slave-zzwf"
server_key: "a65ebd73c827068d125baff62a867ed2"
server_key_iv: "8d385607af717dbcc301594fc53e3edc"
server_key_version: "encryptionkey@0"
I20260620 20:22:40.306125 4931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:40.311583 4931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:40.312328 4931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:40.312593 4931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:40.313481 4931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:40.314981 4931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:40.315102 4931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:40.315228 4931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:40.315316 4931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:40.347201 4931 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:39615
I20260620 20:22:40.347272 5057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:39615 every 8 connection(s)
I20260620 20:22:40.348448 4931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:22:40.349903 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 4931
I20260620 20:22:40.350093 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:22:40.350687 32686 external_mini_cluster.cc:1428] Setting key 8c749759e20d2ca7387185dc00ac54f8
I20260620 20:22:40.353724 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:39153
--builtin_ntp_servers=127.31.235.148:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 20:22:40.357827 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:40.350909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:43429 (local address 127.31.235.190:39153)
0620 20:22:40.351362 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:40.351366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:40.352873 (+ 1507us) server_negotiation.cc:407] Connection header received
0620 20:22:40.355559 (+ 2686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:40.355563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:40.355611 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:40.355702 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:40.357119 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:40.357124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:40.357127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:40.357142 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:40.357182 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:40.357193 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:40.357286 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:40.357551 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 20:22:40.357602 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":229,"threads_started":1}
I20260620 20:22:40.359052 5061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:40.351624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39153 (local address 127.31.235.129:43429)
0620 20:22:40.352692 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:40.352772 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:40.355262 (+ 2490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:40.355958 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:40.355972 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:40.356089 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:40.356774 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:22:40.356780 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:40.356829 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:22:40.356946 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:40.357290 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:40.357302 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:40.357312 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 20:22:40.357604 (+ 292us) client_negotiation.cc:240] Negotiation successful
0620 20:22:40.357854 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":305,"threads_started":1}
I20260620 20:22:40.360481 5058 heartbeater.cc:347] Connected to a master server at 127.31.235.190:39153
I20260620 20:22:40.360805 5058 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:40.361446 5058 heartbeater.cc:511] Master 127.31.235.190:39153 requested a full tablet report, sending...
I20260620 20:22:40.363487 4872 ts_manager.cc:194] Registered new tserver with Master: e1ea2eea52a94b5faad67c2a4e3a653d (127.31.235.129:39615)
W20260620 20:22:40.677868 5062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:40.678201 5062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:40.678373 5062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:40.685576 5062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:40.685712 5062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:40.685899 5062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:22:40.695080 5062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:39153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:40.696381 5062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:40.698117 5062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:40.711150 5070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:40.711349 5067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:40.711333 5068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:40.712419 5062 server_base.cc:1034] running on GCE node
I20260620 20:22:40.713086 5062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:40.714133 5062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:40.715358 5062 hybrid_clock.cc:648] HybridClock initialized: now 1781986960715310 us; error 66 us; skew 500 ppm
I20260620 20:22:40.715691 5062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:22:40.718003 5062 webserver.cc:460] Webserver started at http://127.31.235.130:44465/ using document root <none> and password file <none>
I20260620 20:22:40.718703 5062 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:40.718832 5062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:40.719156 5062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:40.721969 5062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "dbdd10fdbc8349a284b81e539de0d671"
format_stamp: "Formatted at 2026-06-20 20:22:40 on dist-test-slave-zzwf"
server_key: "02f8baffbfc291edd6d7a56e949b12e5"
server_key_iv: "a6b469d6c70b4a6c368e34559c821501"
server_key_version: "encryptionkey@0"
I20260620 20:22:40.722734 5062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "dbdd10fdbc8349a284b81e539de0d671"
format_stamp: "Formatted at 2026-06-20 20:22:40 on dist-test-slave-zzwf"
server_key: "02f8baffbfc291edd6d7a56e949b12e5"
server_key_iv: "a6b469d6c70b4a6c368e34559c821501"
server_key_version: "encryptionkey@0"
I20260620 20:22:40.727643 5062 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:22:40.731170 5076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:40.732574 5062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:22:40.732719 5062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "dbdd10fdbc8349a284b81e539de0d671"
format_stamp: "Formatted at 2026-06-20 20:22:40 on dist-test-slave-zzwf"
server_key: "02f8baffbfc291edd6d7a56e949b12e5"
server_key_iv: "a6b469d6c70b4a6c368e34559c821501"
server_key_version: "encryptionkey@0"
I20260620 20:22:40.732909 5062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:40.738360 5062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:40.739077 5062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:40.739291 5062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:40.740104 5062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:40.741427 5062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:40.741564 5062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:40.741652 5062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:40.741705 5062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:40.774412 5062 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:40125
I20260620 20:22:40.774484 5188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:40125 every 8 connection(s)
I20260620 20:22:40.776032 5062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:22:40.784139 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:40.778735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:55023 (local address 127.31.235.190:39153)
0620 20:22:40.778946 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:40.778952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:40.780459 (+ 1507us) server_negotiation.cc:407] Connection header received
0620 20:22:40.782162 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:40.782166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:40.782219 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:40.782307 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:40.783577 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:40.783580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:40.783583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:40.783596 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:40.783621 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:40.783631 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:40.783718 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:40.783876 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 20:22:40.783924 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:22:40.785027 5191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:40.779355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39153 (local address 127.31.235.130:55023)
0620 20:22:40.780293 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:40.780332 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:40.781924 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:40.782512 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:40.782520 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:40.782608 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:40.783305 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:22:40.783308 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:40.783345 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:22:40.783436 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:40.783712 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:40.783719 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:40.783724 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:22:40.783924 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 20:22:40.784114 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":291,"threads_started":1}
I20260620 20:22:40.785538 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 5062
I20260620 20:22:40.785701 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:22:40.786247 32686 external_mini_cluster.cc:1428] Setting key 28d290d595e8bbc7fcfd8f44beb138cf
I20260620 20:22:40.786474 5189 heartbeater.cc:347] Connected to a master server at 127.31.235.190:39153
I20260620 20:22:40.786777 5189 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:40.787465 5189 heartbeater.cc:511] Master 127.31.235.190:39153 requested a full tablet report, sending...
I20260620 20:22:40.789009 4872 ts_manager.cc:194] Registered new tserver with Master: dbdd10fdbc8349a284b81e539de0d671 (127.31.235.130:40125)
I20260620 20:22:40.789125 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:39153
--builtin_ntp_servers=127.31.235.148:33657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 20:22:41.134884 5192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:41.135191 5192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:41.135355 5192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:41.142894 5192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:41.143035 5192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:41.143216 5192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:22:41.152282 5192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:39153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:41.153666 5192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:41.155413 5192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:41.168740 5197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:41.168784 5200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:41.168694 5198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:41.169859 5192 server_base.cc:1034] running on GCE node
I20260620 20:22:41.170514 5192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:41.171473 5192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:41.172708 5192 hybrid_clock.cc:648] HybridClock initialized: now 1781986961172663 us; error 65 us; skew 500 ppm
I20260620 20:22:41.173038 5192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:22:41.175418 5192 webserver.cc:460] Webserver started at http://127.31.235.131:33619/ using document root <none> and password file <none>
I20260620 20:22:41.176126 5192 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:41.176257 5192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:41.176585 5192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:41.179481 5192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "86b67e4a72c148e2953db6553df2a615"
format_stamp: "Formatted at 2026-06-20 20:22:41 on dist-test-slave-zzwf"
server_key: "94b1c9aacaa31f9a9cb517c21773b995"
server_key_iv: "a715d0771285823296cffec5528f50cd"
server_key_version: "encryptionkey@0"
I20260620 20:22:41.180274 5192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "86b67e4a72c148e2953db6553df2a615"
format_stamp: "Formatted at 2026-06-20 20:22:41 on dist-test-slave-zzwf"
server_key: "94b1c9aacaa31f9a9cb517c21773b995"
server_key_iv: "a715d0771285823296cffec5528f50cd"
server_key_version: "encryptionkey@0"
I20260620 20:22:41.185359 5192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:22:41.189266 5206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:41.190737 5192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:41.190925 5192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "86b67e4a72c148e2953db6553df2a615"
format_stamp: "Formatted at 2026-06-20 20:22:41 on dist-test-slave-zzwf"
server_key: "94b1c9aacaa31f9a9cb517c21773b995"
server_key_iv: "a715d0771285823296cffec5528f50cd"
server_key_version: "encryptionkey@0"
I20260620 20:22:41.191170 5192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:41.196765 5192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:41.197613 5192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:41.197876 5192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:41.198725 5192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:41.200042 5192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:41.200160 5192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:41.200284 5192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:41.200367 5192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:41.232946 5192 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:46199
I20260620 20:22:41.233002 5319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:46199 every 8 connection(s)
I20260620 20:22:41.234278 5192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:22:41.241446 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 5192
I20260620 20:22:41.241681 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:22:41.242300 32686 external_mini_cluster.cc:1428] Setting key be9be380e08935b0b69f3de83d5993bf
I20260620 20:22:41.242543 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.236852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:37633 (local address 127.31.235.190:39153)
0620 20:22:41.237059 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:41.237063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:41.238525 (+ 1462us) server_negotiation.cc:407] Connection header received
0620 20:22:41.240132 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:41.240138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:41.240206 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:41.240318 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:41.241764 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:41.241769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:41.241774 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:41.241794 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:41.241836 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:41.241853 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:41.241997 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:41.242159 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 20:22:41.242219 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:22:41.243376 5322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.237444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39153 (local address 127.31.235.131:37633)
0620 20:22:41.238373 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:41.238409 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:41.239871 (+ 1462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:41.240529 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:41.240537 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:41.240621 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:41.241391 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:22:41.241393 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:41.241428 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:22:41.241563 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:41.241967 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:41.241974 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:41.241980 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 20:22:41.242186 (+ 206us) client_negotiation.cc:240] Negotiation successful
0620 20:22:41.242446 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":259,"threads_started":1}
I20260620 20:22:41.244685 5320 heartbeater.cc:347] Connected to a master server at 127.31.235.190:39153
I20260620 20:22:41.245004 5320 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:41.245641 5320 heartbeater.cc:511] Master 127.31.235.190:39153 requested a full tablet report, sending...
I20260620 20:22:41.246927 4871 ts_manager.cc:194] Registered new tserver with Master: 86b67e4a72c148e2953db6553df2a615 (127.31.235.131:46199)
I20260620 20:22:41.257755 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:22:41.264647 5060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.262370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33032 (local address 127.31.235.190:39153)
0620 20:22:41.262600 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:41.262608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:41.262969 (+ 361us) server_negotiation.cc:407] Connection header received
0620 20:22:41.263098 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:41.263101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:41.263151 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:41.263227 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:41.264067 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:41.264071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:41.264074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:41.264088 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:41.264125 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:41.264137 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:41.264228 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:41.264291 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 20:22:41.264332 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:22:41.268896 4871 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:22:41.275776 4871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:22:41.294255 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.287576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:39615 (local address 127.0.0.1:49914)
0620 20:22:41.288325 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:41.288382 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:41.288523 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:41.290346 (+ 1823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:41.290358 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:41.290505 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:41.291580 (+ 1075us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:22:41.291586 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:41.291622 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:22:41.291727 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:41.293269 (+ 1542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:41.293284 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:41.293312 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 20:22:41.293524 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 20:22:41.293650 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":303,"threads_started":1}
I20260620 20:22:41.295585 5333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.287890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49914 (local address 127.31.235.129:39615)
0620 20:22:41.289682 (+ 1792us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:41.289694 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:41.289733 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:22:41.289814 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:41.289825 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:41.290103 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:41.291935 (+ 1832us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:41.292970 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:41.292982 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:41.293002 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:41.293032 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:41.293097 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:41.293121 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:41.294996 (+ 1875us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:41.295090 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 20:22:41.295211 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1634,"thread_start_us":1482,"threads_started":1}
I20260620 20:22:41.300010 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.288841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:46199 (local address 127.0.0.1:33950)
0620 20:22:41.294525 (+ 5684us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:41.294547 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:41.294637 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:41.298405 (+ 3768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:41.298412 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:41.298445 (+ 33us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:41.298891 (+ 446us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:22:41.298894 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:41.298910 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:22:41.298952 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:41.299589 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:41.299593 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:41.299598 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:22:41.299710 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 20:22:41.299758 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5553,"thread_start_us":8104,"threads_started":1}
I20260620 20:22:41.300791 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.288233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:40125 (local address 127.0.0.1:36502)
0620 20:22:41.289118 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:41.289136 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:41.289217 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:41.297621 (+ 8404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:41.297625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:41.297650 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:41.299754 (+ 2104us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:22:41.299757 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:41.299771 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:22:41.299812 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:41.300224 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:41.300229 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:41.300234 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:22:41.300351 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 20:22:41.300407 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":699,"threads_started":1}
I20260620 20:22:41.301149 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.289556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33950 (local address 127.31.235.131:46199)
0620 20:22:41.297643 (+ 8087us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:41.297653 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:41.297682 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:22:41.297744 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:41.297750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:41.297916 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:41.298392 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:41.299413 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:41.299418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:41.299427 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:41.299438 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:41.299472 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:41.299484 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:41.299607 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:41.300576 (+ 969us) server_negotiation.cc:299] Negotiation successful
0620 20:22:41.300670 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7915,"thread_start_us":7754,"threads_started":1}
I20260620 20:22:41.301530 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.288757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.31.235.130:40125)
0620 20:22:41.296991 (+ 8234us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:41.297001 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:41.297035 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:22:41.297108 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:41.297117 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:41.297358 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:41.297598 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:41.299981 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:41.299991 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:41.300002 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:41.300029 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:41.300083 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:41.300102 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:41.301031 (+ 929us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:41.301099 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 20:22:41.301189 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8051,"thread_start_us":7932,"threads_started":1}
I20260620 20:22:41.301092 4993 tablet_service.cc:1467] Processing CreateTablet for tablet 4c054e1d1c894db7a3b72db8edaf1aa1 (DEFAULT_TABLE table=test-table [id=05bc8a7066bb43e1844178463b034de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:41.305181 4993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c054e1d1c894db7a3b72db8edaf1aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:41.304751 5255 tablet_service.cc:1467] Processing CreateTablet for tablet 4c054e1d1c894db7a3b72db8edaf1aa1 (DEFAULT_TABLE table=test-table [id=05bc8a7066bb43e1844178463b034de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:41.304751 5124 tablet_service.cc:1467] Processing CreateTablet for tablet 4c054e1d1c894db7a3b72db8edaf1aa1 (DEFAULT_TABLE table=test-table [id=05bc8a7066bb43e1844178463b034de9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:41.306100 5255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c054e1d1c894db7a3b72db8edaf1aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:41.306371 5124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c054e1d1c894db7a3b72db8edaf1aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:41.316352 5338 tablet_bootstrap.cc:492] T 4c054e1d1c894db7a3b72db8edaf1aa1 P e1ea2eea52a94b5faad67c2a4e3a653d: Bootstrap starting.
I20260620 20:22:41.320317 5338 tablet_bootstrap.cc:654] T 4c054e1d1c894db7a3b72db8edaf1aa1 P e1ea2eea52a94b5faad67c2a4e3a653d: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:41.320365 5339 tablet_bootstrap.cc:492] T 4c054e1d1c894db7a3b72db8edaf1aa1 P dbdd10fdbc8349a284b81e539de0d671: Bootstrap starting.
I20260620 20:22:41.321722 5338 log.cc:824] T 4c054e1d1c894db7a3b72db8edaf1aa1 P e1ea2eea52a94b5faad67c2a4e3a653d: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:41.322130 5340 tablet_bootstrap.cc:492] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615: Bootstrap starting.
I20260620 20:22:41.324237 5339 tablet_bootstrap.cc:654] T 4c054e1d1c894db7a3b72db8edaf1aa1 P dbdd10fdbc8349a284b81e539de0d671: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:41.324818 5340 tablet_bootstrap.cc:654] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:41.325675 5339 log.cc:824] T 4c054e1d1c894db7a3b72db8edaf1aa1 P dbdd10fdbc8349a284b81e539de0d671: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:41.325866 5340 log.cc:824] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:41.329296 5338 tablet_bootstrap.cc:492] T 4c054e1d1c894db7a3b72db8edaf1aa1 P e1ea2eea52a94b5faad67c2a4e3a653d: No bootstrap required, opened a new log
I20260620 20:22:41.329674 5338 ts_tablet_manager.cc:1397] T 4c054e1d1c894db7a3b72db8edaf1aa1 P e1ea2eea52a94b5faad67c2a4e3a653d: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260620 20:22:41.332042 5340 tablet_bootstrap.cc:492] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615: No bootstrap required, opened a new log
I20260620 20:22:41.332290 5340 ts_tablet_manager.cc:1397] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260620 20:22:41.332321 5339 tablet_bootstrap.cc:492] T 4c054e1d1c894db7a3b72db8edaf1aa1 P dbdd10fdbc8349a284b81e539de0d671: No bootstrap required, opened a new log
I20260620 20:22:41.332535 5339 ts_tablet_manager.cc:1397] T 4c054e1d1c894db7a3b72db8edaf1aa1 P dbdd10fdbc8349a284b81e539de0d671: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20260620 20:22:41.340076 5340 raft_consensus.cc:348] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1ea2eea52a94b5faad67c2a4e3a653d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 39615 } } peers { permanent_uuid: "86b67e4a72c148e2953db6553df2a615" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46199 } } peers { permanent_uuid: "dbdd10fdbc8349a284b81e539de0d671" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40125 } }
I20260620 20:22:41.340535 5340 raft_consensus.cc:374] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:41.340687 5340 raft_consensus.cc:729] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b67e4a72c148e2953db6553df2a615, State: Initialized, Role: FOLLOWER
I20260620 20:22:41.340834 5338 raft_consensus.cc:348] T 4c054e1d1c894db7a3b72db8edaf1aa1 P e1ea2eea52a94b5faad67c2a4e3a653d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1ea2eea52a94b5faad67c2a4e3a653d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 39615 } } peers { permanent_uuid: "86b67e4a72c148e2953db6553df2a615" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46199 } } peers { permanent_uuid: "dbdd10fdbc8349a284b81e539de0d671" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40125 } }
I20260620 20:22:41.341238 5338 raft_consensus.cc:374] T 4c054e1d1c894db7a3b72db8edaf1aa1 P e1ea2eea52a94b5faad67c2a4e3a653d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:41.341336 5338 raft_consensus.cc:729] T 4c054e1d1c894db7a3b72db8edaf1aa1 P e1ea2eea52a94b5faad67c2a4e3a653d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1ea2eea52a94b5faad67c2a4e3a653d, State: Initialized, Role: FOLLOWER
I20260620 20:22:41.341317 5340 consensus_queue.cc:260] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1ea2eea52a94b5faad67c2a4e3a653d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 39615 } } peers { permanent_uuid: "86b67e4a72c148e2953db6553df2a615" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46199 } } peers { permanent_uuid: "dbdd10fdbc8349a284b81e539de0d671" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40125 } }
I20260620 20:22:41.341355 5339 raft_consensus.cc:348] T 4c054e1d1c894db7a3b72db8edaf1aa1 P dbdd10fdbc8349a284b81e539de0d671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1ea2eea52a94b5faad67c2a4e3a653d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 39615 } } peers { permanent_uuid: "86b67e4a72c148e2953db6553df2a615" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46199 } } peers { permanent_uuid: "dbdd10fdbc8349a284b81e539de0d671" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40125 } }
I20260620 20:22:41.341837 5339 raft_consensus.cc:374] T 4c054e1d1c894db7a3b72db8edaf1aa1 P dbdd10fdbc8349a284b81e539de0d671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:41.341990 5339 raft_consensus.cc:729] T 4c054e1d1c894db7a3b72db8edaf1aa1 P dbdd10fdbc8349a284b81e539de0d671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbdd10fdbc8349a284b81e539de0d671, State: Initialized, Role: FOLLOWER
I20260620 20:22:41.341986 5338 consensus_queue.cc:260] T 4c054e1d1c894db7a3b72db8edaf1aa1 P e1ea2eea52a94b5faad67c2a4e3a653d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1ea2eea52a94b5faad67c2a4e3a653d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 39615 } } peers { permanent_uuid: "86b67e4a72c148e2953db6553df2a615" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46199 } } peers { permanent_uuid: "dbdd10fdbc8349a284b81e539de0d671" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40125 } }
I20260620 20:22:41.342548 5320 heartbeater.cc:503] Master 127.31.235.190:39153 was elected leader, sending a full tablet report...
I20260620 20:22:41.343199 5058 heartbeater.cc:503] Master 127.31.235.190:39153 was elected leader, sending a full tablet report...
I20260620 20:22:41.343284 5340 ts_tablet_manager.cc:1428] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260620 20:22:41.343793 5338 ts_tablet_manager.cc:1428] T 4c054e1d1c894db7a3b72db8edaf1aa1 P e1ea2eea52a94b5faad67c2a4e3a653d: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260620 20:22:41.344163 5339 consensus_queue.cc:260] T 4c054e1d1c894db7a3b72db8edaf1aa1 P dbdd10fdbc8349a284b81e539de0d671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1ea2eea52a94b5faad67c2a4e3a653d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 39615 } } peers { permanent_uuid: "86b67e4a72c148e2953db6553df2a615" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46199 } } peers { permanent_uuid: "dbdd10fdbc8349a284b81e539de0d671" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40125 } }
I20260620 20:22:41.345168 5189 heartbeater.cc:503] Master 127.31.235.190:39153 was elected leader, sending a full tablet report...
I20260620 20:22:41.345785 5339 ts_tablet_manager.cc:1428] T 4c054e1d1c894db7a3b72db8edaf1aa1 P dbdd10fdbc8349a284b81e539de0d671: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260620 20:22:41.350677 5059 tablet.cc:2369] T 4c054e1d1c894db7a3b72db8edaf1aa1 P e1ea2eea52a94b5faad67c2a4e3a653d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:22:41.377699 5344 raft_consensus.cc:482] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:22:41.378018 5344 raft_consensus.cc:504] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1ea2eea52a94b5faad67c2a4e3a653d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 39615 } } peers { permanent_uuid: "86b67e4a72c148e2953db6553df2a615" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46199 } } peers { permanent_uuid: "dbdd10fdbc8349a284b81e539de0d671" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40125 } }
I20260620 20:22:41.379673 5344 leader_election.cc:288] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1ea2eea52a94b5faad67c2a4e3a653d (127.31.235.129:39615), dbdd10fdbc8349a284b81e539de0d671 (127.31.235.130:40125)
I20260620 20:22:41.382411 5322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.380203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:40125 (local address 127.31.235.131:56285)
0620 20:22:41.380439 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:41.380455 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:41.380593 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:41.380931 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:41.380936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:41.380965 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:41.381618 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:22:41.381621 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:41.381635 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:22:41.381674 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:41.381990 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:41.381995 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:41.381999 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 20:22:41.382109 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 20:22:41.382155 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260620 20:22:41.382432 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.380107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:56285 (local address 127.31.235.130:40125)
0620 20:22:41.380301 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:41.380307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:41.380550 (+ 243us) server_negotiation.cc:407] Connection header received
0620 20:22:41.380740 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:41.380750 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:41.380799 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:41.380908 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:41.381825 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:41.381830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:41.381834 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:41.381853 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:41.381887 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:41.381898 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:41.382019 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:41.382095 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 20:22:41.382156 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:22:41.382912 5333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.380107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:49115 (local address 127.31.235.129:39615)
0620 20:22:41.380301 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:41.380307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:41.381062 (+ 755us) server_negotiation.cc:407] Connection header received
0620 20:22:41.381224 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:41.381229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:41.381283 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:41.381385 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:41.382312 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:41.382317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:41.382321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:41.382338 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:41.382374 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:41.382385 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:41.382497 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:41.382592 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 20:22:41.382646 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 20:22:41.383040 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.380203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:39615 (local address 127.31.235.131:49115)
0620 20:22:41.380952 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:41.380970 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:41.381097 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:41.381410 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:41.381415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:41.381438 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:41.382140 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:22:41.382143 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:41.382154 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:22:41.382184 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:41.382491 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:41.382496 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:41.382501 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:22:41.382613 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 20:22:41.382656 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":35,"thread_start_us":396,"threads_started":1}
I20260620 20:22:41.383431 5144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c054e1d1c894db7a3b72db8edaf1aa1" candidate_uuid: "86b67e4a72c148e2953db6553df2a615" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbdd10fdbc8349a284b81e539de0d671" is_pre_election: true
I20260620 20:22:41.383981 5144 raft_consensus.cc:2454] T 4c054e1d1c894db7a3b72db8edaf1aa1 P dbdd10fdbc8349a284b81e539de0d671 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86b67e4a72c148e2953db6553df2a615 in term 0.
I20260620 20:22:41.384016 5013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c054e1d1c894db7a3b72db8edaf1aa1" candidate_uuid: "86b67e4a72c148e2953db6553df2a615" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1ea2eea52a94b5faad67c2a4e3a653d" is_pre_election: true
I20260620 20:22:41.384541 5013 raft_consensus.cc:2454] T 4c054e1d1c894db7a3b72db8edaf1aa1 P e1ea2eea52a94b5faad67c2a4e3a653d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86b67e4a72c148e2953db6553df2a615 in term 0.
I20260620 20:22:41.384644 5210 leader_election.cc:302] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86b67e4a72c148e2953db6553df2a615, dbdd10fdbc8349a284b81e539de0d671; no voters:
I20260620 20:22:41.385007 5344 raft_consensus.cc:2781] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:22:41.385174 5344 raft_consensus.cc:482] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:22:41.385279 5344 raft_consensus.cc:3037] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:41.387701 5344 raft_consensus.cc:504] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1ea2eea52a94b5faad67c2a4e3a653d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 39615 } } peers { permanent_uuid: "86b67e4a72c148e2953db6553df2a615" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46199 } } peers { permanent_uuid: "dbdd10fdbc8349a284b81e539de0d671" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40125 } }
I20260620 20:22:41.388267 5344 leader_election.cc:288] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 [CANDIDATE]: Term 1 election: Requested vote from peers e1ea2eea52a94b5faad67c2a4e3a653d (127.31.235.129:39615), dbdd10fdbc8349a284b81e539de0d671 (127.31.235.130:40125)
I20260620 20:22:41.388676 5013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c054e1d1c894db7a3b72db8edaf1aa1" candidate_uuid: "86b67e4a72c148e2953db6553df2a615" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1ea2eea52a94b5faad67c2a4e3a653d"
I20260620 20:22:41.388873 5013 raft_consensus.cc:3037] T 4c054e1d1c894db7a3b72db8edaf1aa1 P e1ea2eea52a94b5faad67c2a4e3a653d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:41.388801 5144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c054e1d1c894db7a3b72db8edaf1aa1" candidate_uuid: "86b67e4a72c148e2953db6553df2a615" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbdd10fdbc8349a284b81e539de0d671"
I20260620 20:22:41.389078 5144 raft_consensus.cc:3037] T 4c054e1d1c894db7a3b72db8edaf1aa1 P dbdd10fdbc8349a284b81e539de0d671 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:41.391103 5013 raft_consensus.cc:2454] T 4c054e1d1c894db7a3b72db8edaf1aa1 P e1ea2eea52a94b5faad67c2a4e3a653d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86b67e4a72c148e2953db6553df2a615 in term 1.
I20260620 20:22:41.391495 5144 raft_consensus.cc:2454] T 4c054e1d1c894db7a3b72db8edaf1aa1 P dbdd10fdbc8349a284b81e539de0d671 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86b67e4a72c148e2953db6553df2a615 in term 1.
I20260620 20:22:41.391496 5207 leader_election.cc:302] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86b67e4a72c148e2953db6553df2a615, e1ea2eea52a94b5faad67c2a4e3a653d; no voters:
I20260620 20:22:41.391919 5344 raft_consensus.cc:2781] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:41.392340 5344 raft_consensus.cc:686] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 [term 1 LEADER]: Becoming Leader. State: Replica: 86b67e4a72c148e2953db6553df2a615, State: Running, Role: LEADER
I20260620 20:22:41.392793 5344 consensus_queue.cc:237] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1ea2eea52a94b5faad67c2a4e3a653d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 39615 } } peers { permanent_uuid: "86b67e4a72c148e2953db6553df2a615" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46199 } } peers { permanent_uuid: "dbdd10fdbc8349a284b81e539de0d671" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40125 } }
I20260620 20:22:41.397876 4870 catalog_manager.cc:5495] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86b67e4a72c148e2953db6553df2a615 (127.31.235.131). New cstate: current_term: 1 leader_uuid: "86b67e4a72c148e2953db6553df2a615" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1ea2eea52a94b5faad67c2a4e3a653d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 39615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86b67e4a72c148e2953db6553df2a615" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 46199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbdd10fdbc8349a284b81e539de0d671" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40125 } health_report { overall_health: UNKNOWN } } }
I20260620 20:22:41.419404 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.417052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33958 (local address 127.31.235.131:46199)
0620 20:22:41.417244 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:41.417250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:41.417423 (+ 173us) server_negotiation.cc:407] Connection header received
0620 20:22:41.417647 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:41.417651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:41.417725 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:41.417819 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:41.418740 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:41.418744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:41.418747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:41.418760 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:41.418789 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:41.418799 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:41.418896 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:41.419064 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 20:22:41.419123 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:22:41.427556 5144 raft_consensus.cc:1261] T 4c054e1d1c894db7a3b72db8edaf1aa1 P dbdd10fdbc8349a284b81e539de0d671 [term 1 FOLLOWER]: Refusing update from remote peer 86b67e4a72c148e2953db6553df2a615: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:41.427556 5013 raft_consensus.cc:1261] T 4c054e1d1c894db7a3b72db8edaf1aa1 P e1ea2eea52a94b5faad67c2a4e3a653d [term 1 FOLLOWER]: Refusing update from remote peer 86b67e4a72c148e2953db6553df2a615: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:41.428579 5348 consensus_queue.cc:1035] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbdd10fdbc8349a284b81e539de0d671" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:41.428974 5344 consensus_queue.cc:1035] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1ea2eea52a94b5faad67c2a4e3a653d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 39615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:41.444672 5352 mvcc.cc:205] Tried to move back new op lower bound from 7299018593997176832 to 7299018593869070336. Current Snapshot: MvccSnapshot[applied={T|T < 7299018593997176832}]
I20260620 20:22:41.456758 4870 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:41.457062 4870 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:41.460827 4870 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09de2f15bb1247beae190ddbf0a0fda5: Sending DeleteTablet for 3 replicas of tablet 4c054e1d1c894db7a3b72db8edaf1aa1
I20260620 20:22:41.462033 5124 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c054e1d1c894db7a3b72db8edaf1aa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:41 UTC) from {username='slave'} at 127.0.0.1:36502
I20260620 20:22:41.462306 4993 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c054e1d1c894db7a3b72db8edaf1aa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:41 UTC) from {username='slave'} at 127.0.0.1:49914
I20260620 20:22:41.462956 5255 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c054e1d1c894db7a3b72db8edaf1aa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:41 UTC) from {username='slave'} at 127.0.0.1:33950
I20260620 20:22:41.463968 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 4931
I20260620 20:22:41.475824 5363 tablet_replica.cc:330] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615: stopping tablet replica
I20260620 20:22:41.477555 5333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.465305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49918 (local address 127.31.235.129:39615)
0620 20:22:41.465474 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:41.465482 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:41.474127 (+ 8645us) server_negotiation.cc:407] Connection header received
0620 20:22:41.474389 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:41.474405 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:41.474543 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:41.474631 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:41.476492 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:41.476501 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:41.476508 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:41.476536 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:41.476586 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:41.476606 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:41.476731 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:41.477004 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 20:22:41.477085 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:22:41.479331 5053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:41.487028 5363 raft_consensus.cc:2229] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:22:41.488072 5363 raft_consensus.cc:2258] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:41.492046 5363 ts_tablet_manager.cc:1905] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:41.502060 5361 tablet_replica.cc:330] T 4c054e1d1c894db7a3b72db8edaf1aa1 P dbdd10fdbc8349a284b81e539de0d671: stopping tablet replica
I20260620 20:22:41.502609 5361 raft_consensus.cc:2229] T 4c054e1d1c894db7a3b72db8edaf1aa1 P dbdd10fdbc8349a284b81e539de0d671 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:22:41.503178 5361 raft_consensus.cc:2258] T 4c054e1d1c894db7a3b72db8edaf1aa1 P dbdd10fdbc8349a284b81e539de0d671 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:41.505631 5361 ts_tablet_manager.cc:1905] T 4c054e1d1c894db7a3b72db8edaf1aa1 P dbdd10fdbc8349a284b81e539de0d671: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:41.505874 5363 ts_tablet_manager.cc:1918] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:22:41.506160 5363 log.cc:1196] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/4c054e1d1c894db7a3b72db8edaf1aa1
I20260620 20:22:41.506850 5363 ts_tablet_manager.cc:1939] T 4c054e1d1c894db7a3b72db8edaf1aa1 P 86b67e4a72c148e2953db6553df2a615: Deleting consensus metadata
I20260620 20:22:41.509088 4856 catalog_manager.cc:4841] TS 86b67e4a72c148e2953db6553df2a615 (127.31.235.131:46199): tablet 4c054e1d1c894db7a3b72db8edaf1aa1 (table test-table [id=05bc8a7066bb43e1844178463b034de9]) successfully deleted
I20260620 20:22:41.513298 5361 ts_tablet_manager.cc:1918] T 4c054e1d1c894db7a3b72db8edaf1aa1 P dbdd10fdbc8349a284b81e539de0d671: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:22:41.513513 5361 log.cc:1196] T 4c054e1d1c894db7a3b72db8edaf1aa1 P dbdd10fdbc8349a284b81e539de0d671: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/4c054e1d1c894db7a3b72db8edaf1aa1
I20260620 20:22:41.514178 5361 ts_tablet_manager.cc:1939] T 4c054e1d1c894db7a3b72db8edaf1aa1 P dbdd10fdbc8349a284b81e539de0d671: Deleting consensus metadata
I20260620 20:22:41.516216 4859 catalog_manager.cc:4841] TS dbdd10fdbc8349a284b81e539de0d671 (127.31.235.130:40125): tablet 4c054e1d1c894db7a3b72db8edaf1aa1 (table test-table [id=05bc8a7066bb43e1844178463b034de9]) successfully deleted
I20260620 20:22:41.587550 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 4931
I20260620 20:22:41.587600 5362 tablet_replica.cc:330] T 4c054e1d1c894db7a3b72db8edaf1aa1 P e1ea2eea52a94b5faad67c2a4e3a653d: stopping tablet replica
I20260620 20:22:41.604678 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 5062
W20260620 20:22:41.604678 4856 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.235.129:39615 (error 108)
I20260620 20:22:41.607411 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.606510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:39615 (local address 127.0.0.1:49928)
0620 20:22:41.607068 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:41.607215 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 20:22:41.607656 4856 catalog_manager.cc:4584] TS e1ea2eea52a94b5faad67c2a4e3a653d (127.31.235.129:39615): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c054e1d1c894db7a3b72db8edaf1aa1: Network error: Client connection negotiation failed: client connection to 127.31.235.129:39615: connect: Connection refused (error 111)
I20260620 20:22:41.608119 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.605644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.31.235.130:40125)
0620 20:22:41.605832 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:41.605838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:41.605883 (+ 45us) server_negotiation.cc:407] Connection header received
0620 20:22:41.606071 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:41.606078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:41.606211 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:41.606330 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:41.607279 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:41.607284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:41.607287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:41.607303 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:41.607333 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:41.607344 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:41.607485 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:41.607764 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 20:22:41.607833 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:22:41.608834 5184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:41.661372 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.660767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:39615 (local address 127.0.0.1:49932)
0620 20:22:41.661028 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:41.661148 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 20:22:41.662472 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.662083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:39615 (local address 127.0.0.1:49934)
0620 20:22:41.662242 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:41.662336 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 20:22:41.696753 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 5062
I20260620 20:22:41.722462 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 5192
I20260620 20:22:41.723217 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.722681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:39615 (local address 127.0.0.1:49942)
0620 20:22:41.722902 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:41.723029 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 20:22:41.724373 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.723939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:39615 (local address 127.0.0.1:49956)
0620 20:22:41.724162 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:41.724247 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 20:22:41.725626 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.723157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.31.235.131:46199)
0620 20:22:41.723323 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:41.723329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:41.723631 (+ 302us) server_negotiation.cc:407] Connection header received
0620 20:22:41.723727 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:41.723733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:41.723793 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:41.723914 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:41.724883 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:41.724891 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:41.724897 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:41.724916 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:41.724962 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:41.724978 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:41.725128 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:41.725197 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 20:22:41.725253 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:22:41.726455 5315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:41.805016 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.804562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:39615 (local address 127.0.0.1:49960)
0620 20:22:41.804762 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:41.804857 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 20:22:41.806171 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:41.805779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:39615 (local address 127.0.0.1:49970)
0620 20:22:41.805999 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:41.806062 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:39615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 20:22:41.850764 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 5192
I20260620 20:22:41.865307 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 4841
I20260620 20:22:41.866598 4902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:41.976781 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 4841
2026-06-20T20:22:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2671 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 20:22:42.020112 32686 mini_oidc.cc:152] Starting JWKS server
I20260620 20:22:42.023020 32686 webserver.cc:460] Webserver started at https://127.0.0.1:34845/ using document root <none> and password file <none>
I20260620 20:22:42.023293 32686 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 20:22:42.023940 32686 webserver.cc:460] Webserver started at http://127.0.0.1:35783/ using document root <none> and password file <none>
2026-06-20T20:22:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:42Z Disabled control of system clock
I20260620 20:22:42.051792 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:35423
--webserver_interface=127.31.235.190
--webserver_port=0
--builtin_ntp_servers=127.31.235.148:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:35423
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34845/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986916608810-32686-0/testchainca.cert with env {}
W20260620 20:22:42.382000 5375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:42.382344 5375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:42.382521 5375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:42.389865 5375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:42.390050 5375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:42.390139 5375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:42.390249 5375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:42.390332 5375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 20:22:42.390422 5375 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 20:22:42.390501 5375 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34845/jwks.json
I20260620 20:22:42.400564 5375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:35423
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:35423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34845/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986916608810-32686-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:42.401932 5375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:42.403182 5375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:42.414975 5383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:42.414992 5380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:42.415107 5381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:42.416234 5375 server_base.cc:1034] running on GCE node
I20260620 20:22:42.417016 5375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:42.418474 5375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:42.419680 5375 hybrid_clock.cc:648] HybridClock initialized: now 1781986962419645 us; error 28 us; skew 500 ppm
I20260620 20:22:42.420049 5375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:22:42.422358 5375 webserver.cc:460] Webserver started at http://127.31.235.190:35649/ using document root <none> and password file <none>
I20260620 20:22:42.423022 5375 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:42.423113 5375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:42.423381 5375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:42.426209 5375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "353382aeabf043b8b28d478929bbd25b"
format_stamp: "Formatted at 2026-06-20 20:22:42 on dist-test-slave-zzwf"
server_key: "d25b9cf4e8d7e7c178e2b0c72cf8068c"
server_key_iv: "5584c7f401ed151a43cd2b507c7f81ff"
server_key_version: "encryptionkey@0"
I20260620 20:22:42.426944 5375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "353382aeabf043b8b28d478929bbd25b"
format_stamp: "Formatted at 2026-06-20 20:22:42 on dist-test-slave-zzwf"
server_key: "d25b9cf4e8d7e7c178e2b0c72cf8068c"
server_key_iv: "5584c7f401ed151a43cd2b507c7f81ff"
server_key_version: "encryptionkey@0"
I20260620 20:22:42.431950 5375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:22:42.435634 5389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:42.437012 5375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:22:42.437165 5375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "353382aeabf043b8b28d478929bbd25b"
format_stamp: "Formatted at 2026-06-20 20:22:42 on dist-test-slave-zzwf"
server_key: "d25b9cf4e8d7e7c178e2b0c72cf8068c"
server_key_iv: "5584c7f401ed151a43cd2b507c7f81ff"
server_key_version: "encryptionkey@0"
I20260620 20:22:42.437352 5375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:42.451165 5375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:42.451927 5375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:42.452193 5375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:42.470181 5375 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:35423
I20260620 20:22:42.470199 5440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:35423 every 8 connection(s)
I20260620 20:22:42.471580 5375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:42.475405 5441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:42.481604 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 5375
I20260620 20:22:42.481828 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:22:42.482612 32686 external_mini_cluster.cc:1428] Setting key f871b6dec2fdcdeb52c89aed06d22ca6
I20260620 20:22:42.486919 5441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b: Bootstrap starting.
I20260620 20:22:42.490003 5441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:42.491008 5441 log.cc:824] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:42.494247 5441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b: No bootstrap required, opened a new log
I20260620 20:22:42.496583 5444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:42.484612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36204 (local address 127.31.235.190:35423)
0620 20:22:42.485652 (+ 1040us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:42.485663 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:42.485706 (+ 43us) server_negotiation.cc:407] Connection header received
0620 20:22:42.487430 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:42.487458 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:42.487901 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:42.488316 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:42.488895 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:42.490243 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:42.491731 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:42.492524 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:42.492704 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:42.493863 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:42.493872 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:42.493890 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:42.494579 (+ 689us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:42.494680 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:42.494713 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:42.494859 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:42.495061 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 20:22:42.495360 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":335,"threads_started":1}
I20260620 20:22:42.500341 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:42.503109 5441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353382aeabf043b8b28d478929bbd25b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35423 } }
I20260620 20:22:42.503539 5441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:42.503645 5441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353382aeabf043b8b28d478929bbd25b, State: Initialized, Role: FOLLOWER
I20260620 20:22:42.504225 5441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353382aeabf043b8b28d478929bbd25b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35423 } }
I20260620 20:22:42.504444 5441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:42.504575 5441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:42.504740 5441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:42.506958 5441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353382aeabf043b8b28d478929bbd25b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35423 } }
I20260620 20:22:42.507452 5441 leader_election.cc:302] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 353382aeabf043b8b28d478929bbd25b; no voters:
I20260620 20:22:42.508018 5441 leader_election.cc:288] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:42.508164 5446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:42.509436 5446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b [term 1 LEADER]: Becoming Leader. State: Replica: 353382aeabf043b8b28d478929bbd25b, State: Running, Role: LEADER
I20260620 20:22:42.510026 5446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353382aeabf043b8b28d478929bbd25b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35423 } }
I20260620 20:22:42.510375 5441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:42.512985 5448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 353382aeabf043b8b28d478929bbd25b. Latest consensus state: current_term: 1 leader_uuid: "353382aeabf043b8b28d478929bbd25b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353382aeabf043b8b28d478929bbd25b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35423 } } }
I20260620 20:22:42.513109 5447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "353382aeabf043b8b28d478929bbd25b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353382aeabf043b8b28d478929bbd25b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35423 } } }
I20260620 20:22:42.513437 5447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:42.513334 5448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:42.514883 5454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:42.519722 5454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:42.528517 5454 catalog_manager.cc:1347] Generated new cluster ID: f0007189514a4cd1860250a5749fb814
I20260620 20:22:42.528630 5454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:42.540998 5454 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:42.542860 5454 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:42.555383 5454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 353382aeabf043b8b28d478929bbd25b: Generated new TSK 0
I20260620 20:22:42.556439 5454 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 20:22:42.608088 5471 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 20:22:42.608283 5471 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 20:22:42.609596 5444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:42.573137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36216 (local address 127.31.235.190:35423)
0620 20:22:42.573347 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:42.573355 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:42.573850 (+ 495us) server_negotiation.cc:407] Connection header received
0620 20:22:42.574150 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:42.574157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:42.574250 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:42.574409 (+ 159us) server_negotiation.cc:226] Negotiated authn=JWT
0620 20:22:42.574898 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:42.576057 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:42.577446 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:42.577973 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:42.578079 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:42.578695 (+ 616us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 20:22:42.608851 (+ 30156us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 20:22:42.609240 (+ 389us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36216: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 20:22:42.609719 5444 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36216: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 20:22:42.611241 5470 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:42.573014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35423 (local address 127.0.0.1:36216)
0620 20:22:42.573747 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:42.573764 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:42.573908 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:42.574439 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:42.574442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:42.574452 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 20:22:42.574757 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:42.574764 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:42.576207 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:42.576212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:42.577303 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:42.577314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:42.578141 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:42.578148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:42.578344 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:42.578442 (+ 98us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 20:22:42.609225 (+ 30783us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 20:22:42.609605 (+ 380us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.235.190:35423: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":356,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.31.235.190:35423: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 20:22:42.612663 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 5375
I20260620 20:22:42.613940 5436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:42.706902 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 5375
2026-06-20T20:22:42Z chronyd exiting
I20260620 20:22:42.757169 32686 test_util.cc:175] -----------------------------------------------
I20260620 20:22:42.757328 32686 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781986916608810-32686-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (742 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 20:22:42.761313 32686 mini_oidc.cc:152] Starting JWKS server
I20260620 20:22:42.763284 32686 webserver.cc:460] Webserver started at https://127.0.0.1:34861/ using document root <none> and password file <none>
I20260620 20:22:42.763537 32686 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 20:22:42.764220 32686 webserver.cc:460] Webserver started at http://127.0.0.1:34969/ using document root <none> and password file <none>
2026-06-20T20:22:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:42Z Disabled control of system clock
I20260620 20:22:42.793442 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:38699
--webserver_interface=127.31.235.190
--webserver_port=0
--builtin_ntp_servers=127.31.235.148:45297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:38699
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34861/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986916608810-32686-0/testchainca.cert with env {}
W20260620 20:22:43.132512 5478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:43.132809 5478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:43.132898 5478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:43.140300 5478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:43.140414 5478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:43.140452 5478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:43.140501 5478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:43.140547 5478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 20:22:43.140600 5478 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 20:22:43.140640 5478 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34861/jwks.json
I20260620 20:22:43.150592 5478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:45297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:38699
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:38699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34861/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986916608810-32686-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:43.151836 5478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:43.153072 5478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:43.164839 5484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:43.164767 5486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:43.165189 5483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:43.166018 5478 server_base.cc:1034] running on GCE node
I20260620 20:22:43.166738 5478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:43.168181 5478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:43.169380 5478 hybrid_clock.cc:648] HybridClock initialized: now 1781986963169337 us; error 33 us; skew 500 ppm
I20260620 20:22:43.169760 5478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:22:43.172008 5478 webserver.cc:460] Webserver started at http://127.31.235.190:41167/ using document root <none> and password file <none>
I20260620 20:22:43.172703 5478 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:43.172793 5478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:43.173156 5478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:43.176028 5478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "e753e3766a72450fa5d688580fd95cbb"
format_stamp: "Formatted at 2026-06-20 20:22:43 on dist-test-slave-zzwf"
server_key: "440fe07d9a2928ebf56246ce33afbb08"
server_key_iv: "8bf52ca7efcafae2112b1d206a076c39"
server_key_version: "encryptionkey@0"
I20260620 20:22:43.176787 5478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "e753e3766a72450fa5d688580fd95cbb"
format_stamp: "Formatted at 2026-06-20 20:22:43 on dist-test-slave-zzwf"
server_key: "440fe07d9a2928ebf56246ce33afbb08"
server_key_iv: "8bf52ca7efcafae2112b1d206a076c39"
server_key_version: "encryptionkey@0"
I20260620 20:22:43.181870 5478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:22:43.185334 5492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:43.186878 5478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:22:43.187098 5478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "e753e3766a72450fa5d688580fd95cbb"
format_stamp: "Formatted at 2026-06-20 20:22:43 on dist-test-slave-zzwf"
server_key: "440fe07d9a2928ebf56246ce33afbb08"
server_key_iv: "8bf52ca7efcafae2112b1d206a076c39"
server_key_version: "encryptionkey@0"
I20260620 20:22:43.187325 5478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:43.212369 5478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:43.213227 5478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:43.213625 5478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:43.231988 5478 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:38699
I20260620 20:22:43.232028 5543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:38699 every 8 connection(s)
I20260620 20:22:43.233347 5478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:43.237381 5544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:43.243379 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 5478
I20260620 20:22:43.243604 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:22:43.244504 32686 external_mini_cluster.cc:1428] Setting key 6e25ca57b00302c1df486ce419859122
I20260620 20:22:43.251135 5544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb: Bootstrap starting.
I20260620 20:22:43.256367 5544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:43.257930 5544 log.cc:824] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:43.262034 5544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb: No bootstrap required, opened a new log
I20260620 20:22:43.267769 5547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:43.246752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.31.235.190:38699)
0620 20:22:43.247878 (+ 1126us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:43.247896 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:43.247949 (+ 53us) server_negotiation.cc:407] Connection header received
0620 20:22:43.250987 (+ 3038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:43.251012 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:43.251545 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:43.252102 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:43.252642 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:43.255480 (+ 2838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:43.258066 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:43.259450 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:43.259663 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:43.261852 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:43.261860 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:43.261875 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:43.262752 (+ 877us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:43.262866 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:43.262899 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:43.263184 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:43.265921 (+ 2737us) server_negotiation.cc:299] Negotiation successful
0620 20:22:43.266325 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":336,"threads_started":1}
I20260620 20:22:43.272233 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:43.274621 5544 raft_consensus.cc:348] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e753e3766a72450fa5d688580fd95cbb" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 38699 } }
I20260620 20:22:43.275043 5544 raft_consensus.cc:374] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:43.275170 5544 raft_consensus.cc:729] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e753e3766a72450fa5d688580fd95cbb, State: Initialized, Role: FOLLOWER
I20260620 20:22:43.275919 5544 consensus_queue.cc:260] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e753e3766a72450fa5d688580fd95cbb" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 38699 } }
I20260620 20:22:43.276223 5544 raft_consensus.cc:388] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:43.276403 5544 raft_consensus.cc:482] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:43.276625 5544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:43.279433 5544 raft_consensus.cc:504] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e753e3766a72450fa5d688580fd95cbb" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 38699 } }
I20260620 20:22:43.280165 5544 leader_election.cc:302] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e753e3766a72450fa5d688580fd95cbb; no voters:
I20260620 20:22:43.280941 5544 leader_election.cc:288] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:43.281066 5549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:43.281889 5549 raft_consensus.cc:686] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb [term 1 LEADER]: Becoming Leader. State: Replica: e753e3766a72450fa5d688580fd95cbb, State: Running, Role: LEADER
I20260620 20:22:43.282445 5549 consensus_queue.cc:237] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e753e3766a72450fa5d688580fd95cbb" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 38699 } }
I20260620 20:22:43.283401 5544 sys_catalog.cc:564] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:43.285115 5550 sys_catalog.cc:455] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e753e3766a72450fa5d688580fd95cbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e753e3766a72450fa5d688580fd95cbb" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 38699 } } }
I20260620 20:22:43.285557 5550 sys_catalog.cc:458] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:43.286198 5551 sys_catalog.cc:455] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e753e3766a72450fa5d688580fd95cbb. Latest consensus state: current_term: 1 leader_uuid: "e753e3766a72450fa5d688580fd95cbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e753e3766a72450fa5d688580fd95cbb" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 38699 } } }
I20260620 20:22:43.286584 5551 sys_catalog.cc:458] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:43.290097 5554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:43.296514 5554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:43.305904 5554 catalog_manager.cc:1347] Generated new cluster ID: f1a20340094c447d977f40669179f453
I20260620 20:22:43.306044 5554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:43.328018 5554 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:43.329895 5554 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:43.339514 5554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e753e3766a72450fa5d688580fd95cbb: Generated new TSK 0
I20260620 20:22:43.340569 5554 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 20:22:43.383435 5574 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 20:22:43.383602 5574 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 20:22:43.384800 5547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:43.364364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54242 (local address 127.31.235.190:38699)
0620 20:22:43.364540 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:43.364546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:43.364923 (+ 377us) server_negotiation.cc:407] Connection header received
0620 20:22:43.365228 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:43.365234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:43.365339 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:43.365523 (+ 184us) server_negotiation.cc:226] Negotiated authn=JWT
0620 20:22:43.366051 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:43.367152 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:43.368537 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:43.369219 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:43.369337 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:43.369942 (+ 605us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 20:22:43.384129 (+ 14187us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 20:22:43.384509 (+ 380us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54242: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 20:22:43.384919 5547 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54242: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 20:22:43.385025 5573 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:43.364228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:38699 (local address 127.0.0.1:54242)
0620 20:22:43.364827 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:43.364842 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:43.364994 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:43.365600 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:43.365604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:43.365614 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0620 20:22:43.365873 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:43.365881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:43.367310 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:43.367313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:43.368386 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:43.368397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:43.369402 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:43.369407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:43.369632 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:43.369665 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 20:22:43.384392 (+ 14727us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 20:22:43.384675 (+ 283us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.235.190:38699: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":318,"threads_started":1}
I20260620 20:22:43.386088 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 5478
I20260620 20:22:43.387488 5539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:43.480289 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 5478
2026-06-20T20:22:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (774 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 20:22:43.535689 32686 mini_oidc.cc:152] Starting JWKS server
I20260620 20:22:43.537676 32686 webserver.cc:460] Webserver started at https://127.0.0.1:37413/ using document root <none> and password file <none>
I20260620 20:22:43.537982 32686 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 20:22:43.538627 32686 webserver.cc:460] Webserver started at http://127.0.0.1:37601/ using document root <none> and password file <none>
2026-06-20T20:22:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:43Z Disabled control of system clock
I20260620 20:22:43.565263 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:41737
--webserver_interface=127.31.235.190
--webserver_port=0
--builtin_ntp_servers=127.31.235.148:41129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:41737
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37413/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 20:22:43.885080 5581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:43.885367 5581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:43.885454 5581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:43.892805 5581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:43.892918 5581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:43.892958 5581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:43.893007 5581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:43.893052 5581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 20:22:43.893106 5581 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 20:22:43.893146 5581 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37413/jwks.json
I20260620 20:22:43.902417 5581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:41129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:41737
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:41737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37413/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:43.903647 5581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:43.905365 5581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:43.916867 5586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:43.916950 5589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:43.916988 5587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:43.918581 5581 server_base.cc:1034] running on GCE node
I20260620 20:22:43.919379 5581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:43.920794 5581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:43.922029 5581 hybrid_clock.cc:648] HybridClock initialized: now 1781986963921987 us; error 43 us; skew 500 ppm
I20260620 20:22:43.922402 5581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:22:43.924654 5581 webserver.cc:460] Webserver started at http://127.31.235.190:42413/ using document root <none> and password file <none>
I20260620 20:22:43.925302 5581 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:43.925386 5581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:43.925712 5581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:43.928443 5581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "1686dfc4d34641aab891d10725443a57"
format_stamp: "Formatted at 2026-06-20 20:22:43 on dist-test-slave-zzwf"
server_key: "46e8d4a96600a3a95162cf9558ac2239"
server_key_iv: "6dffb4c0a6ab3ab2515da21ba3c2ca0d"
server_key_version: "encryptionkey@0"
I20260620 20:22:43.929168 5581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "1686dfc4d34641aab891d10725443a57"
format_stamp: "Formatted at 2026-06-20 20:22:43 on dist-test-slave-zzwf"
server_key: "46e8d4a96600a3a95162cf9558ac2239"
server_key_iv: "6dffb4c0a6ab3ab2515da21ba3c2ca0d"
server_key_version: "encryptionkey@0"
I20260620 20:22:43.934314 5581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:22:43.937662 5595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:43.939162 5581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:22:43.939301 5581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "1686dfc4d34641aab891d10725443a57"
format_stamp: "Formatted at 2026-06-20 20:22:43 on dist-test-slave-zzwf"
server_key: "46e8d4a96600a3a95162cf9558ac2239"
server_key_iv: "6dffb4c0a6ab3ab2515da21ba3c2ca0d"
server_key_version: "encryptionkey@0"
I20260620 20:22:43.939481 5581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:43.961032 5581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:43.961920 5581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:43.962179 5581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:43.979200 5581 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:41737
I20260620 20:22:43.979224 5646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:41737 every 8 connection(s)
I20260620 20:22:43.980538 5581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:43.983808 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 5581
I20260620 20:22:43.984076 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:22:43.984362 5647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:43.984740 32686 external_mini_cluster.cc:1428] Setting key 6cc2fe834c2a89837b48e5bf72860813
I20260620 20:22:43.996766 5647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57: Bootstrap starting.
I20260620 20:22:43.997817 5650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:43.986611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56182 (local address 127.31.235.190:41737)
0620 20:22:43.987619 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:43.987630 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:43.987675 (+ 45us) server_negotiation.cc:407] Connection header received
0620 20:22:43.989233 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:43.989254 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:43.989786 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:43.990210 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:43.990767 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:43.991995 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:43.993275 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:43.994130 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:43.994358 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:43.995546 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:43.995553 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:43.995566 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:43.996085 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:43.996157 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:43.996177 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:43.996309 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:43.996519 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 20:22:43.996819 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":371,"threads_started":1}
I20260620 20:22:44.000384 5647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:44.001670 5647 log.cc:824] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:44.001929 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:44.004868 5647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57: No bootstrap required, opened a new log
I20260620 20:22:44.012455 5647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1686dfc4d34641aab891d10725443a57" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 41737 } }
I20260620 20:22:44.012784 5647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:44.012918 5647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1686dfc4d34641aab891d10725443a57, State: Initialized, Role: FOLLOWER
I20260620 20:22:44.013622 5647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1686dfc4d34641aab891d10725443a57" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 41737 } }
I20260620 20:22:44.013892 5647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:44.014004 5647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:44.014127 5647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:44.016136 5647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1686dfc4d34641aab891d10725443a57" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 41737 } }
I20260620 20:22:44.016584 5647 leader_election.cc:302] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1686dfc4d34641aab891d10725443a57; no voters:
I20260620 20:22:44.017218 5647 leader_election.cc:288] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:44.017416 5652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:44.018296 5652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57 [term 1 LEADER]: Becoming Leader. State: Replica: 1686dfc4d34641aab891d10725443a57, State: Running, Role: LEADER
I20260620 20:22:44.018865 5652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1686dfc4d34641aab891d10725443a57" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 41737 } }
I20260620 20:22:44.019381 5647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:44.022128 5653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1686dfc4d34641aab891d10725443a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1686dfc4d34641aab891d10725443a57" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 41737 } } }
I20260620 20:22:44.022190 5654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1686dfc4d34641aab891d10725443a57. Latest consensus state: current_term: 1 leader_uuid: "1686dfc4d34641aab891d10725443a57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1686dfc4d34641aab891d10725443a57" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 41737 } } }
I20260620 20:22:44.022670 5653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:44.022723 5654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:44.026841 5659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:44.032585 5659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:44.041643 5659 catalog_manager.cc:1347] Generated new cluster ID: a8c66ecb4a2b42e99282d4bdd8e36b9d
I20260620 20:22:44.041787 5659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:44.059906 5659 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:44.061354 5659 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:44.076126 5659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1686dfc4d34641aab891d10725443a57: Generated new TSK 0
I20260620 20:22:44.077019 5659 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:44.108296 5650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:44.099770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56196 (local address 127.31.235.190:41737)
0620 20:22:44.100009 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:44.100016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:44.100371 (+ 355us) server_negotiation.cc:407] Connection header received
0620 20:22:44.100615 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:44.100621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:44.100696 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:44.100819 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0620 20:22:44.101350 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:44.102239 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:44.104208 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:44.104662 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:44.104765 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:44.105357 (+ 592us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 20:22:44.107691 (+ 2334us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37413/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 20:22:44.108009 (+ 318us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56196: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37413/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 20:22:44.108412 5650 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56196: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37413/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 20:22:44.110512 5676 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:44.099669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:41737 (local address 127.0.0.1:56196)
0620 20:22:44.100258 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:44.100281 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:44.100419 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:44.100865 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:44.100871 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:44.100884 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0620 20:22:44.101202 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:44.101215 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:44.102378 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:44.102384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:44.104063 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:44.104077 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:44.104862 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:44.104870 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:44.105063 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:44.105108 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 20:22:44.107896 (+ 2788us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37413/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 20:22:44.110014 (+ 2118us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.235.190:41737: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37413/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":1844,"thread_start_us":316,"threads_started":1}
I20260620 20:22:44.112617 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 5581
I20260620 20:22:44.113759 5642 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 20:22:44.120275 5677 webserver.cc:533] Webserver: sslize failed:
E20260620 20:22:44.120385 5677 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 20:22:44.206666 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 5581
2026-06-20T20:22:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (726 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T20:22:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:44Z Disabled control of system clock
I20260620 20:22:44.287823 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:34143
--webserver_interface=127.31.235.190
--webserver_port=0
--builtin_ntp_servers=127.31.235.148:45393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:34143
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986916608810-32686-0/testchainca.cert with env {}
W20260620 20:22:44.607636 5682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:44.607935 5682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:44.608111 5682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:44.615840 5682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:44.615972 5682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:44.616041 5682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:44.616147 5682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:44.616235 5682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 20:22:44.616325 5682 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 20:22:44.616386 5682 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 20:22:44.626240 5682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:45393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:34143
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:34143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986916608810-32686-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:44.627547 5682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:44.628815 5682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:44.640731 5687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:44.640828 5688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:44.640779 5690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:44.642903 5682 server_base.cc:1034] running on GCE node
I20260620 20:22:44.643688 5682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:44.645057 5682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:44.646291 5682 hybrid_clock.cc:648] HybridClock initialized: now 1781986964646250 us; error 49 us; skew 500 ppm
I20260620 20:22:44.646641 5682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:22:44.648864 5682 webserver.cc:460] Webserver started at http://127.31.235.190:36241/ using document root <none> and password file <none>
I20260620 20:22:44.649629 5682 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:44.649760 5682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:44.650074 5682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:44.652884 5682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "d4b6f2e816ed441fbd0e66103ffbde27"
format_stamp: "Formatted at 2026-06-20 20:22:44 on dist-test-slave-zzwf"
server_key: "eb9c50590b3bea6fed106637ce92e858"
server_key_iv: "f28a3b3cad038f8ec4f041b0fff05a87"
server_key_version: "encryptionkey@0"
I20260620 20:22:44.653748 5682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "d4b6f2e816ed441fbd0e66103ffbde27"
format_stamp: "Formatted at 2026-06-20 20:22:44 on dist-test-slave-zzwf"
server_key: "eb9c50590b3bea6fed106637ce92e858"
server_key_iv: "f28a3b3cad038f8ec4f041b0fff05a87"
server_key_version: "encryptionkey@0"
I20260620 20:22:44.658723 5682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:22:44.662266 5696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:44.663585 5682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:22:44.663777 5682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "d4b6f2e816ed441fbd0e66103ffbde27"
format_stamp: "Formatted at 2026-06-20 20:22:44 on dist-test-slave-zzwf"
server_key: "eb9c50590b3bea6fed106637ce92e858"
server_key_iv: "f28a3b3cad038f8ec4f041b0fff05a87"
server_key_version: "encryptionkey@0"
I20260620 20:22:44.664044 5682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:44.691233 5682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:44.692121 5682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:44.692412 5682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:44.710635 5682 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:34143
I20260620 20:22:44.710667 5747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:34143 every 8 connection(s)
I20260620 20:22:44.711979 5682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:44.715726 5748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:44.717389 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 5682
I20260620 20:22:44.717618 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:22:44.718439 32686 external_mini_cluster.cc:1428] Setting key c1b67a732111c045c73a4c1de4b8c272
I20260620 20:22:44.728926 5748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27: Bootstrap starting.
I20260620 20:22:44.731468 5751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:44.720394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53342 (local address 127.31.235.190:34143)
0620 20:22:44.721482 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:44.721567 (+ 85us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:44.721613 (+ 46us) server_negotiation.cc:407] Connection header received
0620 20:22:44.723125 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:44.723145 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:44.723600 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:44.723976 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:44.724501 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:44.725839 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:44.727094 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:44.727761 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:44.727972 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:44.729194 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:44.729202 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:44.729219 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:22:44.729803 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:44.729884 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:22:44.729905 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:44.730036 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:44.730259 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 20:22:44.730573 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":448,"threads_started":1}
I20260620 20:22:44.732422 5748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:44.733536 5748 log.cc:824] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:44.735277 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:22:44.736485 5748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27: No bootstrap required, opened a new log
I20260620 20:22:44.744166 5748 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b6f2e816ed441fbd0e66103ffbde27" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 34143 } }
I20260620 20:22:44.744565 5748 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:44.744674 5748 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b6f2e816ed441fbd0e66103ffbde27, State: Initialized, Role: FOLLOWER
I20260620 20:22:44.745293 5748 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b6f2e816ed441fbd0e66103ffbde27" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 34143 } }
I20260620 20:22:44.745597 5748 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:44.745752 5748 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:44.745960 5748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:44.748030 5748 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b6f2e816ed441fbd0e66103ffbde27" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 34143 } }
I20260620 20:22:44.748538 5748 leader_election.cc:302] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4b6f2e816ed441fbd0e66103ffbde27; no voters:
I20260620 20:22:44.749171 5748 leader_election.cc:288] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:44.749357 5753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:44.750684 5753 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27 [term 1 LEADER]: Becoming Leader. State: Replica: d4b6f2e816ed441fbd0e66103ffbde27, State: Running, Role: LEADER
I20260620 20:22:44.751286 5753 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b6f2e816ed441fbd0e66103ffbde27" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 34143 } }
I20260620 20:22:44.751715 5748 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:44.754458 5754 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4b6f2e816ed441fbd0e66103ffbde27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b6f2e816ed441fbd0e66103ffbde27" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 34143 } } }
I20260620 20:22:44.754434 5755 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4b6f2e816ed441fbd0e66103ffbde27. Latest consensus state: current_term: 1 leader_uuid: "d4b6f2e816ed441fbd0e66103ffbde27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4b6f2e816ed441fbd0e66103ffbde27" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 34143 } } }
I20260620 20:22:44.754878 5754 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:44.754999 5755 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:44.758664 5758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:44.764417 5758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:44.773913 5758 catalog_manager.cc:1347] Generated new cluster ID: 52b55eb55e2746639a1caa38fb313951
I20260620 20:22:44.774055 5758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:44.790858 5758 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:44.792687 5758 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:44.802479 5758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4b6f2e816ed441fbd0e66103ffbde27: Generated new TSK 0
I20260620 20:22:44.803504 5758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:44.902314 5751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:44.827492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53358 (local address 127.31.235.190:34143)
0620 20:22:44.827753 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:44.827762 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:44.828196 (+ 434us) server_negotiation.cc:407] Connection header received
0620 20:22:44.828473 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:44.828479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:44.828558 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:44.828705 (+ 147us) server_negotiation.cc:226] Negotiated authn=JWT
0620 20:22:44.829222 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:44.830031 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:44.832090 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:44.832522 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:44.832631 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:44.833192 (+ 561us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 20:22:44.901723 (+ 68531us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 20:22:44.902082 (+ 359us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53358: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 20:22:44.902474 5751 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53358: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 20:22:44.902774 5777 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:44.827466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:34143 (local address 127.0.0.1:53358)
0620 20:22:44.828069 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:44.828090 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:44.828242 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:44.828751 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:44.828759 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:44.828775 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0620 20:22:44.829071 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:44.829082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:44.830166 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:44.830172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:44.831917 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:44.831931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:44.832697 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:44.832703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:44.832892 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:44.832947 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 20:22:44.902044 (+ 69097us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 20:22:44.902363 (+ 319us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.235.190:34143: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":308,"threads_started":1}
I20260620 20:22:44.903859 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 5682
I20260620 20:22:44.905099 5743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:45.006151 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 5682
2026-06-20T20:22:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (780 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (427 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (412 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:45 dist-test-slave-zzwf krb5kdc[5792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:45 dist-test-slave-zzwf krb5kdc[5792](info): set up 2 sockets
Jun 20 20:22:45 dist-test-slave-zzwf krb5kdc[5792](info): commencing operation
krb5kdc: starting...
W20260620 20:22:46.938635 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:22:47 dist-test-slave-zzwf krb5kdc[5792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:22:47.153963 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:33243
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:39599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:33243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:47.482390 5805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:47.482725 5805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:47.482872 5805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:47.490204 5805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:47.490316 5805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:47.490401 5805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:47.490514 5805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:47.490577 5805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:47.500671 5805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:39599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:33243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:33243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:47.502081 5805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:47.503515 5805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:47.515658 5813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:47.515698 5811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:47.515633 5810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:47.516415 5805 server_base.cc:1034] running on GCE node
I20260620 20:22:47.517171 5805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:47.518525 5805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:47.519716 5805 hybrid_clock.cc:648] HybridClock initialized: now 1781986967519660 us; error 51 us; skew 500 ppm
I20260620 20:22:47.520007 5805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:47 dist-test-slave-zzwf krb5kdc[5792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986967, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:47.522977 5805 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:22:47.524466 5805 webserver.cc:460] Webserver started at http://127.31.235.190:34429/ using document root <none> and password file <none>
I20260620 20:22:47.525102 5805 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:47.525198 5805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:47.525437 5805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:47.528343 5805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "e7e4efa11e90457e8c285feeb026a3ad"
format_stamp: "Formatted at 2026-06-20 20:22:47 on dist-test-slave-zzwf"
server_key: "a4ed7ad771d3399e1675071ee42fbdc5"
server_key_iv: "2279633d39ff1ba0fffc367ac86a92f2"
server_key_version: "encryptionkey@0"
I20260620 20:22:47.529048 5805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "e7e4efa11e90457e8c285feeb026a3ad"
format_stamp: "Formatted at 2026-06-20 20:22:47 on dist-test-slave-zzwf"
server_key: "a4ed7ad771d3399e1675071ee42fbdc5"
server_key_iv: "2279633d39ff1ba0fffc367ac86a92f2"
server_key_version: "encryptionkey@0"
I20260620 20:22:47.534168 5805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:22:47.537569 5820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:47.538899 5805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:22:47.539068 5805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "e7e4efa11e90457e8c285feeb026a3ad"
format_stamp: "Formatted at 2026-06-20 20:22:47 on dist-test-slave-zzwf"
server_key: "a4ed7ad771d3399e1675071ee42fbdc5"
server_key_iv: "2279633d39ff1ba0fffc367ac86a92f2"
server_key_version: "encryptionkey@0"
I20260620 20:22:47.539283 5805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:47.553335 5805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:47.557366 5805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:47.557693 5805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:47.576301 5805 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:33243
I20260620 20:22:47.576320 5871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:33243 every 8 connection(s)
I20260620 20:22:47.577840 5805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:47.581437 5872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:47.584610 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 5805
I20260620 20:22:47.584816 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:22:47.585623 32686 external_mini_cluster.cc:1428] Setting key 8ec750fd5bf913b43c5f2d34ce0597ef
I20260620 20:22:47.593483 5872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad: Bootstrap starting.
I20260620 20:22:47.596606 5872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:47.597653 5872 log.cc:824] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:22:47 dist-test-slave-zzwf krb5kdc[5792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:47.600818 5872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad: No bootstrap required, opened a new log
I20260620 20:22:47.607700 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:47.587677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42968 (local address 127.31.235.190:33243)
0620 20:22:47.588398 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:47.588409 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:47.588448 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:22:47.590139 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:47.590161 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:47.590547 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:47.590943 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:47.592329 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:47.593689 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:47.594960 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:47.595624 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:47.595811 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:47.600358 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:47.600365 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:47.600401 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:47.600439 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:47.603599 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:47.604394 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:47.604401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:47.604407 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:47.604556 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:47.605071 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:47.605074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:47.605076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:47.605635 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:47.605857 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:47.606314 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 20:22:47.606677 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":310,"threads_started":1}
I20260620 20:22:47.610193 5872 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e4efa11e90457e8c285feeb026a3ad" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33243 } }
I20260620 20:22:47.610651 5872 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:47.610773 5872 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e4efa11e90457e8c285feeb026a3ad, State: Initialized, Role: FOLLOWER
I20260620 20:22:47.611289 5872 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e4efa11e90457e8c285feeb026a3ad" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33243 } }
I20260620 20:22:47.611529 5872 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:47.611675 5872 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:47.611845 5872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:47.614051 5872 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e4efa11e90457e8c285feeb026a3ad" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33243 } }
I20260620 20:22:47.614583 5872 leader_election.cc:302] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7e4efa11e90457e8c285feeb026a3ad; no voters:
I20260620 20:22:47.615245 5872 leader_election.cc:288] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:47.615347 5877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:47.615967 5877 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad [term 1 LEADER]: Becoming Leader. State: Replica: e7e4efa11e90457e8c285feeb026a3ad, State: Running, Role: LEADER
I20260620 20:22:47.616539 5877 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e4efa11e90457e8c285feeb026a3ad" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33243 } }
I20260620 20:22:47.617408 5872 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:47.619385 5879 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7e4efa11e90457e8c285feeb026a3ad. Latest consensus state: current_term: 1 leader_uuid: "e7e4efa11e90457e8c285feeb026a3ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e4efa11e90457e8c285feeb026a3ad" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33243 } } }
I20260620 20:22:47.619784 5879 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:47.624730 5878 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7e4efa11e90457e8c285feeb026a3ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e4efa11e90457e8c285feeb026a3ad" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33243 } } }
I20260620 20:22:47.625149 5878 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:47.627465 5886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:47.632364 5886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:47.641566 5886 catalog_manager.cc:1347] Generated new cluster ID: 18769ae41fc441c59eaa1f6d78691820
I20260620 20:22:47.641685 5886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:47.650349 5886 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:47.652144 5886 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:47.672467 5886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7e4efa11e90457e8c285feeb026a3ad: Generated new TSK 0
I20260620 20:22:47.673610 5886 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
I20260620 20:22:47.729730 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33243
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:39599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:48.050171 5898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:48.050438 5898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:48.050531 5898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:48.058236 5898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:48.058344 5898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:48.058465 5898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:22:48.068019 5898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:39599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:48.069198 5898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:48.070449 5898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:48.083741 5903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:48.083660 5906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:48.083678 5904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:48.084479 5898 server_base.cc:1034] running on GCE node
I20260620 20:22:48.085084 5898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:48.086174 5898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:48.087426 5898 hybrid_clock.cc:648] HybridClock initialized: now 1781986968087371 us; error 47 us; skew 500 ppm
I20260620 20:22:48.087781 5898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:48 dist-test-slave-zzwf krb5kdc[5792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986968, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:48.091044 5898 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:22:48.092526 5898 webserver.cc:460] Webserver started at http://127.31.235.129:35229/ using document root <none> and password file <none>
I20260620 20:22:48.093225 5898 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:48.093370 5898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:48.093765 5898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:48.096700 5898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "aa6ed1835c6b4c0db88775cbe356201b"
format_stamp: "Formatted at 2026-06-20 20:22:48 on dist-test-slave-zzwf"
server_key: "8aa4d035dc79de79b701e3aaf05ee0ac"
server_key_iv: "e0f248fe90c6654ca57d7f5111465807"
server_key_version: "encryptionkey@0"
I20260620 20:22:48.097577 5898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "aa6ed1835c6b4c0db88775cbe356201b"
format_stamp: "Formatted at 2026-06-20 20:22:48 on dist-test-slave-zzwf"
server_key: "8aa4d035dc79de79b701e3aaf05ee0ac"
server_key_iv: "e0f248fe90c6654ca57d7f5111465807"
server_key_version: "encryptionkey@0"
I20260620 20:22:48.102945 5898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260620 20:22:48.106395 5913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:48.107750 5898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 20:22:48.107926 5898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "aa6ed1835c6b4c0db88775cbe356201b"
format_stamp: "Formatted at 2026-06-20 20:22:48 on dist-test-slave-zzwf"
server_key: "8aa4d035dc79de79b701e3aaf05ee0ac"
server_key_iv: "e0f248fe90c6654ca57d7f5111465807"
server_key_version: "encryptionkey@0"
I20260620 20:22:48.108139 5898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:48.147812 5898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:48.152532 5898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:48.152861 5898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:48.153918 5898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:48.155313 5898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:48.155416 5898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:48.155570 5898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:48.155652 5898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:48.188845 5898 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:44909
I20260620 20:22:48.188898 6025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:44909 every 8 connection(s)
I20260620 20:22:48.190205 5898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:22:48.199920 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 5898
I20260620 20:22:48.200125 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:22:48.200762 32686 external_mini_cluster.cc:1428] Setting key a08efa1ff653f4539d2bc980da74ca86
Jun 20 20:22:48 dist-test-slave-zzwf krb5kdc[5792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986968, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:48.209437 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:48.192695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:50427 (local address 127.31.235.190:33243)
0620 20:22:48.193133 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:48.193138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:48.193932 (+ 794us) server_negotiation.cc:407] Connection header received
0620 20:22:48.195722 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:48.195729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:48.195825 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:48.195940 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:48.197354 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:48.198139 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:48.199675 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:48.200117 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:48.200260 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:48.203903 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:48.203908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:48.203911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:48.203941 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:48.206496 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:48.207220 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:48.207227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:48.207230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:48.207347 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:48.207759 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:48.207766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:48.207769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:48.208023 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:48.208175 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:48.208858 (+ 683us) server_negotiation.cc:299] Negotiation successful
0620 20:22:48.209073 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":241,"threads_started":1}
I20260620 20:22:48.210461 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:48.193032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33243 (local address 127.31.235.129:50427)
0620 20:22:48.193770 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:48.193813 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:48.195457 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:48.196163 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:48.196171 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:48.196663 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:48.197192 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:48.197208 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:48.198298 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:48.198308 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:48.199526 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:48.199538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:48.200385 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:48.200392 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:48.200615 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:48.201313 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:48.201335 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:48.203677 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:48.206645 (+ 2968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:48.206654 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:48.206672 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:48.207087 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:48.207457 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:48.207462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:48.207465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:48.207651 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:48.208163 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:48.208170 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:48.208502 (+ 332us) client_negotiation.cc:769] Sending connection context
0620 20:22:48.208769 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 20:22:48.209314 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":268,"threads_started":1}
I20260620 20:22:48.212074 6026 heartbeater.cc:347] Connected to a master server at 127.31.235.190:33243
I20260620 20:22:48.212404 6026 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:48.213304 6026 heartbeater.cc:511] Master 127.31.235.190:33243 requested a full tablet report, sending...
I20260620 20:22:48.216007 5837 ts_manager.cc:194] Registered new tserver with Master: aa6ed1835c6b4c0db88775cbe356201b (127.31.235.129:44909)
I20260620 20:22:48.218336 5837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:50427
WARNING: no policy specified for kudu/127.31.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
I20260620 20:22:48.249846 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33243
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:39599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:48.596997 6032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:48.597297 6032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:48.597469 6032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:48.605813 6032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:48.605942 6032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:48.606137 6032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:22:48.616556 6032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:39599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:48.618106 6032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:48.619550 6032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:48.633265 6037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:48.633306 6040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:48.633523 6038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:48.634620 6032 server_base.cc:1034] running on GCE node
I20260620 20:22:48.635244 6032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:48.636180 6032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:48.637382 6032 hybrid_clock.cc:648] HybridClock initialized: now 1781986968637342 us; error 28 us; skew 500 ppm
I20260620 20:22:48.637743 6032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:48 dist-test-slave-zzwf krb5kdc[5792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986968, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:48.641119 6032 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:22:48.642805 6032 webserver.cc:460] Webserver started at http://127.31.235.130:46371/ using document root <none> and password file <none>
I20260620 20:22:48.643520 6032 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:48.643664 6032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:48.643977 6032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:48.646916 6032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "35aaeee23e5b4e29ae57ee860aa20144"
format_stamp: "Formatted at 2026-06-20 20:22:48 on dist-test-slave-zzwf"
server_key: "b80f551fc6601de196e3af834a9c7ca2"
server_key_iv: "592688a10a71d55c70f5bea60d056f4f"
server_key_version: "encryptionkey@0"
I20260620 20:22:48.647755 6032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "35aaeee23e5b4e29ae57ee860aa20144"
format_stamp: "Formatted at 2026-06-20 20:22:48 on dist-test-slave-zzwf"
server_key: "b80f551fc6601de196e3af834a9c7ca2"
server_key_iv: "592688a10a71d55c70f5bea60d056f4f"
server_key_version: "encryptionkey@0"
I20260620 20:22:48.652912 6032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:22:48.656448 6047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:48.657981 6032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:48.658164 6032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "35aaeee23e5b4e29ae57ee860aa20144"
format_stamp: "Formatted at 2026-06-20 20:22:48 on dist-test-slave-zzwf"
server_key: "b80f551fc6601de196e3af834a9c7ca2"
server_key_iv: "592688a10a71d55c70f5bea60d056f4f"
server_key_version: "encryptionkey@0"
I20260620 20:22:48.658365 6032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:48.708196 6032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:48.712919 6032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:48.713233 6032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:48.714320 6032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:48.715665 6032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:48.715771 6032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:48.715924 6032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:48.715996 6032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:48.750082 6032 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:45553
I20260620 20:22:48.750140 6159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:45553 every 8 connection(s)
I20260620 20:22:48.751365 6032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:22:48.753952 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 6032
I20260620 20:22:48.754127 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:22:48.754716 32686 external_mini_cluster.cc:1428] Setting key 92257f35ec4a37cbbcc985a960b65688
Jun 20 20:22:48 dist-test-slave-zzwf krb5kdc[5792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986968, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:48.772845 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:48.754011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:48191 (local address 127.31.235.190:33243)
0620 20:22:48.754529 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:48.754536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:48.755533 (+ 997us) server_negotiation.cc:407] Connection header received
0620 20:22:48.757340 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:48.757348 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:48.757441 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:48.757586 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:48.759052 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:48.760187 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:48.761844 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:48.762564 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:48.762720 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:48.767130 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:48.767141 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:48.767144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:48.767177 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:48.769796 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:48.770508 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:48.770516 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:48.770521 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:48.770711 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:48.771145 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:48.771150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:48.771153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:48.771406 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:48.771568 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:48.772253 (+ 685us) server_negotiation.cc:299] Negotiation successful
0620 20:22:48.772470 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":260,"threads_started":1}
I20260620 20:22:48.773952 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:48.754586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33243 (local address 127.31.235.130:48191)
0620 20:22:48.755364 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:48.755421 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:48.757079 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:48.757775 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:48.757782 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:48.758314 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:48.758868 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:48.758884 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:48.760350 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:48.760355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:48.761708 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:48.761720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:48.762837 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:48.762845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:48.763254 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:48.764055 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:48.764093 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:48.766877 (+ 2784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:48.769939 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:48.769948 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:48.769966 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:48.770385 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:48.770836 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:48.770841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:48.770845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:48.771029 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:48.771543 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:48.771551 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:48.771907 (+ 356us) client_negotiation.cc:769] Sending connection context
0620 20:22:48.772179 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 20:22:48.772725 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":352,"threads_started":1}
WARNING: no policy specified for kudu/127.31.235.131@KRBTEST.COM; defaulting to no policy
I20260620 20:22:48.775404 6160 heartbeater.cc:347] Connected to a master server at 127.31.235.190:33243
I20260620 20:22:48.775709 6160 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:48.776409 6160 heartbeater.cc:511] Master 127.31.235.190:33243 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.131@KRBTEST.COM" created.
I20260620 20:22:48.777817 5837 ts_manager.cc:194] Registered new tserver with Master: 35aaeee23e5b4e29ae57ee860aa20144 (127.31.235.130:45553)
I20260620 20:22:48.778654 5837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:48191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
I20260620 20:22:48.801043 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33243
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:39599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:49.128338 6166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:49.128619 6166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:49.128712 6166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:49.135637 6166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:49.135736 6166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:49.135843 6166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:22:49.145443 6166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:39599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:49.146591 6166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:49.147765 6166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:49.161573 6174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:49.161679 6172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:49.161644 6171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:49.163049 6166 server_base.cc:1034] running on GCE node
I20260620 20:22:49.163645 6166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:49.164482 6166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:49.165690 6166 hybrid_clock.cc:648] HybridClock initialized: now 1781986969165653 us; error 58 us; skew 500 ppm
I20260620 20:22:49.166051 6166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:49 dist-test-slave-zzwf krb5kdc[5792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986969, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:49.169080 6166 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:22:49.170804 6166 webserver.cc:460] Webserver started at http://127.31.235.131:33895/ using document root <none> and password file <none>
I20260620 20:22:49.171634 6166 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:49.171782 6166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:49.172101 6166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:49.175159 6166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "7ef8c5cc7dc5443ca11bffc701312360"
format_stamp: "Formatted at 2026-06-20 20:22:49 on dist-test-slave-zzwf"
server_key: "4ce8c91eb3f04c94c400360a1d3da8b7"
server_key_iv: "fe515487247ed7f12faa034b35e1454c"
server_key_version: "encryptionkey@0"
I20260620 20:22:49.175954 6166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "7ef8c5cc7dc5443ca11bffc701312360"
format_stamp: "Formatted at 2026-06-20 20:22:49 on dist-test-slave-zzwf"
server_key: "4ce8c91eb3f04c94c400360a1d3da8b7"
server_key_iv: "fe515487247ed7f12faa034b35e1454c"
server_key_version: "encryptionkey@0"
I20260620 20:22:49.180961 6166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:22:49.184374 6181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:49.185806 6166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:22:49.186008 6166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "7ef8c5cc7dc5443ca11bffc701312360"
format_stamp: "Formatted at 2026-06-20 20:22:49 on dist-test-slave-zzwf"
server_key: "4ce8c91eb3f04c94c400360a1d3da8b7"
server_key_iv: "fe515487247ed7f12faa034b35e1454c"
server_key_version: "encryptionkey@0"
I20260620 20:22:49.186228 6166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:49.208323 6166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:49.212321 6166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:49.212646 6166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:49.213591 6166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:49.214834 6166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:49.214907 6166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:49.214989 6166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:49.215044 6166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:49.222244 6026 heartbeater.cc:503] Master 127.31.235.190:33243 was elected leader, sending a full tablet report...
I20260620 20:22:49.248140 6166 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:36149
I20260620 20:22:49.248191 6293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:36149 every 8 connection(s)
I20260620 20:22:49.249372 6166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:22:49.251916 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 6166
I20260620 20:22:49.252089 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:22:49.252508 32686 external_mini_cluster.cc:1428] Setting key 66c2e33499da66beee2a1c203717829d
Jun 20 20:22:49 dist-test-slave-zzwf krb5kdc[5792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986969, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:22:49.268373 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:49.251876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:40615 (local address 127.31.235.190:33243)
0620 20:22:49.252097 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:49.252113 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:49.253465 (+ 1352us) server_negotiation.cc:407] Connection header received
0620 20:22:49.255368 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:49.255373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:49.255455 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:49.255575 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:49.256991 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:49.257815 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:49.259291 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:49.259692 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:49.259829 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:49.263120 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:49.263124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:49.263128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:49.263159 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:49.265482 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:49.266174 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:49.266178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:49.266180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:49.266286 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:49.266703 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:49.266710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:49.266714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:49.266993 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:49.267179 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:49.267839 (+ 660us) server_negotiation.cc:299] Negotiation successful
0620 20:22:49.268060 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:22:49.269320 6296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:49.252432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33243 (local address 127.31.235.131:40615)
0620 20:22:49.253316 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:49.253359 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:49.255110 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:49.255804 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:49.255812 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:49.256281 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:49.256830 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:49.256845 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:49.257954 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:49.257960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:49.259142 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:49.259153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:49.259905 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:49.259911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:49.260141 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:49.260748 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:49.260769 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:49.262939 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:49.265640 (+ 2701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:49.265646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:49.265658 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:49.266045 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:49.266399 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:49.266406 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:49.266409 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:49.266593 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:49.267165 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:49.267172 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:49.267469 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 20:22:49.267743 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 20:22:49.268235 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":387,"threads_started":1}
I20260620 20:22:49.270503 6294 heartbeater.cc:347] Connected to a master server at 127.31.235.190:33243
I20260620 20:22:49.270771 6294 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:49.271389 6294 heartbeater.cc:511] Master 127.31.235.190:33243 requested a full tablet report, sending...
I20260620 20:22:49.272578 5837 ts_manager.cc:194] Registered new tserver with Master: 7ef8c5cc7dc5443ca11bffc701312360 (127.31.235.131:36149)
I20260620 20:22:49.273325 5837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:40615
I20260620 20:22:49.279455 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/test-admin.keytab.
Jun 20 20:22:49 dist-test-slave-zzwf krb5kdc[5792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:49.309415 32686 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 20:22:49.316162 6301 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:49.312797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34320 (local address 127.31.235.129:44909)
0620 20:22:49.314370 (+ 1573us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:49.314378 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:49.314414 (+ 36us) server_negotiation.cc:407] Connection header received
0620 20:22:49.314483 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:49.314489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:49.314694 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:49.314826 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:49.315679 (+ 853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34320: BlockingRecv error: recv got EOF from 127.0.0.1:34320 (error 108)
Metrics: {"server-negotiator.queue_time_us":1425,"thread_start_us":1278,"threads_started":1}
W20260620 20:22:49.316262 32686 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/krb5cc' of type FILE
Jun 20 20:22:49 dist-test-slave-zzwf krb5kdc[5792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:49.317941 32686 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 20:22:49 dist-test-slave-zzwf krb5kdc[5792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:49.319562 32686 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 20:22:49 dist-test-slave-zzwf krb5kdc[5792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:22:49.334690 6301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:49.321232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.31.235.129:44909)
0620 20:22:49.321448 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:49.321453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:49.321726 (+ 273us) server_negotiation.cc:407] Connection header received
0620 20:22:49.321856 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:49.321859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:49.321916 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:49.321993 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:49.322896 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:49.323836 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:49.325239 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:49.325913 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:49.326010 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:49.329403 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:49.329415 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:49.329441 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:49.329477 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:49.332001 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:49.332651 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:49.332660 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:49.332663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:49.332789 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:49.333145 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:49.333150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:49.333153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:49.333482 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:49.333845 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:49.334187 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 20:22:49.334426 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260620 20:22:49.340405 6301 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:49.338653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.31.235.129:44909)
0620 20:22:49.338865 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:49.338871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:49.339105 (+ 234us) server_negotiation.cc:407] Connection header received
0620 20:22:49.339298 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:49.339303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:49.339371 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:49.339474 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:49.340150 (+ 676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34348: BlockingRecv error: recv got EOF from 127.0.0.1:34348 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 20:22:49.340765 32686 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/krb5cc' of type FILE
Jun 20 20:22:49 dist-test-slave-zzwf krb5kdc[5792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:49.342161 32686 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 20:22:49 dist-test-slave-zzwf krb5kdc[5792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:49.343523 32686 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 20:22:49 dist-test-slave-zzwf krb5kdc[5792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:22:49.356711 6301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:49.344761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.31.235.129:44909)
0620 20:22:49.345015 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:49.345020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:49.345174 (+ 154us) server_negotiation.cc:407] Connection header received
0620 20:22:49.345253 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:49.345256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:49.345308 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:49.345386 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:49.346286 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:49.346989 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:49.348119 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:49.348547 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:49.348637 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:49.352118 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:49.352123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:49.352126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:49.352165 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:49.354535 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:49.355192 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:49.355196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:49.355198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:49.355308 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:49.355617 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:49.355620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:49.355622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:49.355860 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:49.356073 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:49.356239 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 20:22:49.356382 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 20:22:49.360741 6301 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:22:49.359162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34372 (local address 127.31.235.129:44909)
0620 20:22:49.359365 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:49.359370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:49.359559 (+ 189us) server_negotiation.cc:407] Connection header received
0620 20:22:49.359671 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:49.359675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:49.359734 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:49.359807 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:49.360536 (+ 729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34372: BlockingRecv error: recv got EOF from 127.0.0.1:34372 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 20:22:49.360814 32686 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781986916608810-32686-0/krb5kdc/krb5cc' of type FILE
Jun 20 20:22:49 dist-test-slave-zzwf krb5kdc[5792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:49.362399 32686 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 20:22:49 dist-test-slave-zzwf krb5kdc[5792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:49.363758 32686 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 20:22:49 dist-test-slave-zzwf krb5kdc[5792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:22:49.376657 6301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:49.364951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34382 (local address 127.31.235.129:44909)
0620 20:22:49.365137 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:49.365143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:49.365355 (+ 212us) server_negotiation.cc:407] Connection header received
0620 20:22:49.365460 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:49.365464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:49.365587 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:49.365661 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:49.366513 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:49.367237 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:49.368347 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:49.368750 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:49.368841 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:49.372068 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:49.372072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:49.372075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:49.372101 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:49.374464 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:49.375045 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:49.375049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:49.375051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:49.375168 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:49.375479 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:49.375483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:49.375485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:49.375754 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:49.376022 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:49.376140 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 20:22:49.376294 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:22:49.378258 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 5898
I20260620 20:22:49.390405 6301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:49.379013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34384 (local address 127.31.235.129:44909)
0620 20:22:49.379167 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:49.379171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:49.379565 (+ 394us) server_negotiation.cc:407] Connection header received
0620 20:22:49.379723 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:49.379727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:49.379780 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:49.379860 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:49.380876 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:49.381675 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:49.382977 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:49.383392 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:49.383486 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:49.385571 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:49.385576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:49.385579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:49.385607 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:49.387946 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:49.388557 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:49.388561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:49.388563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:49.388678 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:49.389048 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:49.389051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:49.389053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:49.389280 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:49.389462 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:49.389913 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 20:22:49.390064 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:22:49.391017 6021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:49.473435 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 5898
I20260620 20:22:49.489809 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 6032
Jun 20 20:22:49 dist-test-slave-zzwf krb5kdc[5792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.130@KRBTEST.COM
I20260620 20:22:49.505218 6314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:49.490861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.31.235.130:45553)
0620 20:22:49.491413 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:49.491421 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:49.491446 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:22:49.491504 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:49.491511 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:49.491761 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:49.491936 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:49.492948 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:49.494041 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:49.495260 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:49.495788 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:49.495940 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:49.499691 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:49.499701 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:49.499715 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:49.499745 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:49.502501 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:49.503168 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:49.503184 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:49.503186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:49.503302 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:49.503661 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:49.503667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:49.503670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:49.504018 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:49.504319 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:49.504647 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 20:22:49.504886 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":243,"threads_started":1}
I20260620 20:22:49.506556 6155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:49.588579 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 6032
I20260620 20:22:49.604157 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 6166
Jun 20 20:22:49 dist-test-slave-zzwf krb5kdc[5792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.131@KRBTEST.COM
I20260620 20:22:49.619298 6316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:49.605056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36398 (local address 127.31.235.131:36149)
0620 20:22:49.605674 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:49.605681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:49.605711 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:22:49.605770 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:49.605776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:49.605975 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:49.606114 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:49.607097 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:49.608094 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:49.609292 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:49.609978 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:49.610082 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:49.613803 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:49.613813 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:49.613828 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:49.613862 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:49.616457 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:49.617165 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:49.617172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:49.617174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:49.617292 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:49.617843 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:49.617846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:49.617848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:49.618264 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:49.618487 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:49.618793 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 20:22:49.618992 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":281,"threads_started":1}
I20260620 20:22:49.620645 6289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:49.702998 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 6166
I20260620 20:22:49.719265 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 5805
I20260620 20:22:49.720623 5867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:49.823211 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 5805
2026-06-20T20:22:49Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3982 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:49 dist-test-slave-zzwf krb5kdc[6321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:49 dist-test-slave-zzwf krb5kdc[6321](info): set up 2 sockets
Jun 20 20:22:49 dist-test-slave-zzwf krb5kdc[6321](info): commencing operation
krb5kdc: starting...
W20260620 20:22:50.922689 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:22:51 dist-test-slave-zzwf krb5kdc[6321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:22:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:22:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.190.keytab.
Entry for principal oryx/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.190.keytab.
I20260620 20:22:51.148574 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:33833
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.190.keytab
--principal=oryx/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:33833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:51.466104 6334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:51.466382 6334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:51.466564 6334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:51.473677 6334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:51.473783 6334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:51.473871 6334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:51.473964 6334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:51.474025 6334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:51.483589 6334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:33833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.190.keytab
--principal=oryx/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:33833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:51.484942 6334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:51.486265 6334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:51.498027 6342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:51.498027 6339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:51.498021 6340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:51.498838 6334 server_base.cc:1034] running on GCE node
I20260620 20:22:51.499580 6334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:51.500963 6334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:51.502197 6334 hybrid_clock.cc:648] HybridClock initialized: now 1781986971502158 us; error 55 us; skew 500 ppm
I20260620 20:22:51.502542 6334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:51 dist-test-slave-zzwf krb5kdc[6321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986971, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:51.505707 6334 init.cc:377] Logged in from keytab as oryx/127.31.235.190@KRBTEST.COM (short username oryx)
I20260620 20:22:51.507248 6334 webserver.cc:460] Webserver started at http://127.31.235.190:34863/ using document root <none> and password file <none>
I20260620 20:22:51.507901 6334 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:51.508042 6334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:51.508345 6334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:51.511258 6334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "4efb3f7a9871466ca1c95f235aafdb86"
format_stamp: "Formatted at 2026-06-20 20:22:51 on dist-test-slave-zzwf"
server_key: "3ef63d542491da51294abf498f7fe0f1"
server_key_iv: "da164e1eaead442162531e5659130ad6"
server_key_version: "encryptionkey@0"
I20260620 20:22:51.512004 6334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "4efb3f7a9871466ca1c95f235aafdb86"
format_stamp: "Formatted at 2026-06-20 20:22:51 on dist-test-slave-zzwf"
server_key: "3ef63d542491da51294abf498f7fe0f1"
server_key_iv: "da164e1eaead442162531e5659130ad6"
server_key_version: "encryptionkey@0"
I20260620 20:22:51.517112 6334 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:22:51.520612 6349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:51.522092 6334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:51.522272 6334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "4efb3f7a9871466ca1c95f235aafdb86"
format_stamp: "Formatted at 2026-06-20 20:22:51 on dist-test-slave-zzwf"
server_key: "3ef63d542491da51294abf498f7fe0f1"
server_key_iv: "da164e1eaead442162531e5659130ad6"
server_key_version: "encryptionkey@0"
I20260620 20:22:51.522477 6334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:51.543453 6334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:51.547983 6334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:51.548290 6334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:51.566829 6334 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:33833
I20260620 20:22:51.566861 6400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:33833 every 8 connection(s)
I20260620 20:22:51.568192 6334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:22:51.571723 6401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:51.578573 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 6334
I20260620 20:22:51.578763 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:22:51.579524 32686 external_mini_cluster.cc:1428] Setting key 14dc177e0ebbf07b03609563a555cadb
I20260620 20:22:51.581732 6401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86: Bootstrap starting.
I20260620 20:22:51.585752 6401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:51.586795 6401 log.cc:824] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:51.589813 6401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86: No bootstrap required, opened a new log
Jun 20 20:22:51 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM
I20260620 20:22:51.599607 6401 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } }
I20260620 20:22:51.600064 6401 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:51.600167 6401 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4efb3f7a9871466ca1c95f235aafdb86, State: Initialized, Role: FOLLOWER
I20260620 20:22:51.600863 6401 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } }
I20260620 20:22:51.601131 6401 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:22:51.601291 6401 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:22:51.601511 6401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:51.603745 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:51.581455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35276 (local address 127.31.235.190:33833)
0620 20:22:51.582286 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:51.582302 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:51.582349 (+ 47us) server_negotiation.cc:407] Connection header received
0620 20:22:51.584746 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:51.584771 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:51.585237 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:51.585674 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:51.586934 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:51.588711 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:51.590062 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:51.591082 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:51.591252 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:51.595232 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:51.595242 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:51.595261 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:51.595308 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:51.599187 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:51.599851 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:51.599860 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:51.599867 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:51.600075 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:51.600518 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:51.600523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:51.600526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:51.601214 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:51.601564 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:51.601945 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 20:22:51.602400 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":322,"threads_started":1}
I20260620 20:22:51.604605 6401 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } }
I20260620 20:22:51.605089 6401 leader_election.cc:302] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4efb3f7a9871466ca1c95f235aafdb86; no voters:
I20260620 20:22:51.605724 6401 leader_election.cc:288] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:22:51.605883 6406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:51.607645 6401 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:51.607656 6406 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [term 1 LEADER]: Becoming Leader. State: Replica: 4efb3f7a9871466ca1c95f235aafdb86, State: Running, Role: LEADER
I20260620 20:22:51.608184 6406 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } }
I20260620 20:22:51.611126 6408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4efb3f7a9871466ca1c95f235aafdb86. Latest consensus state: current_term: 1 leader_uuid: "4efb3f7a9871466ca1c95f235aafdb86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } }
I20260620 20:22:51.611461 6408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:51.611950 6407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4efb3f7a9871466ca1c95f235aafdb86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } }
I20260620 20:22:51.612255 6407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:51.612625 6416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:51.617237 6416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:51.625094 6416 catalog_manager.cc:1347] Generated new cluster ID: 66a41b8acf634ce197e6bc9780a4c844
I20260620 20:22:51.625208 6416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:51.641047 6416 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:22:51.642462 6416 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:51.650022 6416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86: Generated new TSK 0
I20260620 20:22:51.650838 6416 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.129.keytab.
Entry for principal oryx/127.31.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.129.keytab.
I20260620 20:22:51.709555 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33833
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.129.keytab
--principal=oryx/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:52.051949 6427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:52.052237 6427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:52.052389 6427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:52.059953 6427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:52.060108 6427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:52.060285 6427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:22:52.070705 6427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.129.keytab
--principal=oryx/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:52.072058 6427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:52.073316 6427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:52.086655 6435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:52.086614 6433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:52.086681 6432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:52.087963 6427 server_base.cc:1034] running on GCE node
I20260620 20:22:52.088611 6427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:52.089600 6427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:52.090827 6427 hybrid_clock.cc:648] HybridClock initialized: now 1781986972090771 us; error 49 us; skew 500 ppm
I20260620 20:22:52.091168 6427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:52 dist-test-slave-zzwf krb5kdc[6321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986972, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:52.094204 6427 init.cc:377] Logged in from keytab as oryx/127.31.235.129@KRBTEST.COM (short username oryx)
I20260620 20:22:52.095543 6427 webserver.cc:460] Webserver started at http://127.31.235.129:42717/ using document root <none> and password file <none>
I20260620 20:22:52.096199 6427 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:52.096340 6427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:52.096643 6427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:52.099529 6427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "838a74503a5f46b39c7553da21104e3d"
format_stamp: "Formatted at 2026-06-20 20:22:52 on dist-test-slave-zzwf"
server_key: "10815dffd9b021d32a6238e1cb99074c"
server_key_iv: "ca381059cdc74893a21ee1fa5fbe65da"
server_key_version: "encryptionkey@0"
I20260620 20:22:52.100293 6427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "838a74503a5f46b39c7553da21104e3d"
format_stamp: "Formatted at 2026-06-20 20:22:52 on dist-test-slave-zzwf"
server_key: "10815dffd9b021d32a6238e1cb99074c"
server_key_iv: "ca381059cdc74893a21ee1fa5fbe65da"
server_key_version: "encryptionkey@0"
I20260620 20:22:52.105285 6427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:22:52.108788 6442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:52.110332 6427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:52.110493 6427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "838a74503a5f46b39c7553da21104e3d"
format_stamp: "Formatted at 2026-06-20 20:22:52 on dist-test-slave-zzwf"
server_key: "10815dffd9b021d32a6238e1cb99074c"
server_key_iv: "ca381059cdc74893a21ee1fa5fbe65da"
server_key_version: "encryptionkey@0"
I20260620 20:22:52.110728 6427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:52.135537 6427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:52.139858 6427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:52.140187 6427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:52.141208 6427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:52.142652 6427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:52.142724 6427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:52.142864 6427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:52.142933 6427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:52.175931 6427 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:37795
I20260620 20:22:52.175988 6554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:37795 every 8 connection(s)
I20260620 20:22:52.177232 6427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:22:52.183048 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 6427
I20260620 20:22:52.183271 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:22:52.183883 32686 external_mini_cluster.cc:1428] Setting key 3aab77d5f39a0bf9004812cbe1b32d66
Jun 20 20:22:52 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986972, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM
I20260620 20:22:52.198624 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:52.179831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:54127 (local address 127.31.235.190:33833)
0620 20:22:52.180347 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:52.180352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:52.181341 (+ 989us) server_negotiation.cc:407] Connection header received
0620 20:22:52.183188 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:52.183194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:52.183280 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:52.183410 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:52.185069 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:52.185991 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:52.187506 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:52.187894 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:52.188030 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:52.192113 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:52.192119 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:52.192123 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:52.192159 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:52.195058 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:52.195939 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:52.195945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:52.195949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:52.196076 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:52.196525 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:52.196531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:52.196546 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:52.196814 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:52.196968 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:52.197797 (+ 829us) server_negotiation.cc:299] Negotiation successful
0620 20:22:52.198074 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":259,"threads_started":1}
I20260620 20:22:52.199189 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:52.180401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.129:54127)
0620 20:22:52.181094 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:52.181131 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:52.182919 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:52.183672 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:52.183681 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:52.184232 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:52.184907 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:52.184924 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:52.186152 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:52.186161 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:52.187346 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:52.187357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:52.188113 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:52.188122 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:52.188363 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:52.189178 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:52.189200 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:52.191827 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:52.195306 (+ 3479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:52.195323 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:52.195345 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:52.195799 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:52.196186 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:52.196192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:52.196196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:52.196389 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:52.196946 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:52.196955 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:52.197330 (+ 375us) client_negotiation.cc:769] Sending connection context
0620 20:22:52.197641 (+ 311us) client_negotiation.cc:240] Negotiation successful
0620 20:22:52.198057 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":263,"threads_started":1}
I20260620 20:22:52.200815 6555 heartbeater.cc:347] Connected to a master server at 127.31.235.190:33833
I20260620 20:22:52.201211 6555 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:52.202243 6555 heartbeater.cc:511] Master 127.31.235.190:33833 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.31.235.130@KRBTEST.COM; defaulting to no policy
I20260620 20:22:52.204493 6366 ts_manager.cc:194] Registered new tserver with Master: 838a74503a5f46b39c7553da21104e3d (127.31.235.129:37795)
I20260620 20:22:52.205991 6366 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:54127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.130.keytab.
Entry for principal oryx/127.31.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.130.keytab.
I20260620 20:22:52.233954 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33833
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.130.keytab
--principal=oryx/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:52.555148 6561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:52.555419 6561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:52.555588 6561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:52.562888 6561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:52.562994 6561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:52.563225 6561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:22:52.572770 6561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.130.keytab
--principal=oryx/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:52.573999 6561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:52.575224 6561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:52.587814 6566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:52.587780 6569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:52.587733 6567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:52.588747 6561 server_base.cc:1034] running on GCE node
I20260620 20:22:52.589350 6561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:52.590334 6561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:52.591558 6561 hybrid_clock.cc:648] HybridClock initialized: now 1781986972591515 us; error 57 us; skew 500 ppm
I20260620 20:22:52.591873 6561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:52 dist-test-slave-zzwf krb5kdc[6321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986972, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:52.594920 6561 init.cc:377] Logged in from keytab as oryx/127.31.235.130@KRBTEST.COM (short username oryx)
I20260620 20:22:52.596256 6561 webserver.cc:460] Webserver started at http://127.31.235.130:44173/ using document root <none> and password file <none>
I20260620 20:22:52.596843 6561 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:52.596937 6561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:52.597168 6561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:52.599931 6561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "b23046502b304dbc98a013e3192761b5"
format_stamp: "Formatted at 2026-06-20 20:22:52 on dist-test-slave-zzwf"
server_key: "a098732dd6cf8d54c7cf3aff517b637f"
server_key_iv: "5159278bf4bc664c9696ac52c028dc72"
server_key_version: "encryptionkey@0"
I20260620 20:22:52.600629 6561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "b23046502b304dbc98a013e3192761b5"
format_stamp: "Formatted at 2026-06-20 20:22:52 on dist-test-slave-zzwf"
server_key: "a098732dd6cf8d54c7cf3aff517b637f"
server_key_iv: "5159278bf4bc664c9696ac52c028dc72"
server_key_version: "encryptionkey@0"
I20260620 20:22:52.605369 6561 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260620 20:22:52.608734 6576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:52.610155 6561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:52.610280 6561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "b23046502b304dbc98a013e3192761b5"
format_stamp: "Formatted at 2026-06-20 20:22:52 on dist-test-slave-zzwf"
server_key: "a098732dd6cf8d54c7cf3aff517b637f"
server_key_iv: "5159278bf4bc664c9696ac52c028dc72"
server_key_version: "encryptionkey@0"
I20260620 20:22:52.610447 6561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:52.623832 6561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:52.627975 6561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:52.628221 6561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:52.629093 6561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:52.630432 6561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:52.630503 6561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:52.630584 6561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:52.630633 6561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:52.664124 6561 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:39559
I20260620 20:22:52.664183 6688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:39559 every 8 connection(s)
I20260620 20:22:52.665395 6561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:22:52.673796 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 6561
I20260620 20:22:52.674036 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:22:52.674542 32686 external_mini_cluster.cc:1428] Setting key 8ab25907fce5a77eede510d57b514955
Jun 20 20:22:52 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986972, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM
I20260620 20:22:52.687129 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:52.668062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:36989 (local address 127.31.235.190:33833)
0620 20:22:52.668243 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:52.668248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:52.669461 (+ 1213us) server_negotiation.cc:407] Connection header received
0620 20:22:52.671176 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:52.671181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:52.671248 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:52.671344 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:52.673401 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:52.674534 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:52.676579 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:52.676996 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:52.677133 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:52.680871 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:52.680876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:52.680879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:52.680913 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:52.683561 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:52.684601 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:52.684605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:52.684607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:52.684738 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:52.685252 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:52.685255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:52.685257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:52.685617 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:52.685739 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:52.686668 (+ 929us) server_negotiation.cc:299] Negotiation successful
0620 20:22:52.686869 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:22:52.688652 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:52.668579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.130:36989)
0620 20:22:52.669290 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:52.669326 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:52.670911 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:52.671576 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:52.671591 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:52.672470 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:52.673180 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:52.673198 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:52.674763 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:52.674770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:52.676435 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:52.676447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:52.677240 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:52.677248 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:52.677474 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:52.678288 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:52.678313 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:52.680612 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:52.683741 (+ 3129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:52.683754 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:52.683777 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:52.684457 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:52.684855 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:52.684863 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:52.684871 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:52.685133 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:52.685741 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:52.685752 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:52.686237 (+ 485us) client_negotiation.cc:769] Sending connection context
0620 20:22:52.686547 (+ 310us) client_negotiation.cc:240] Negotiation successful
0620 20:22:52.687152 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":275,"threads_started":1}
I20260620 20:22:52.690346 6689 heartbeater.cc:347] Connected to a master server at 127.31.235.190:33833
I20260620 20:22:52.690675 6689 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:52.691430 6689 heartbeater.cc:511] Master 127.31.235.190:33833 requested a full tablet report, sending...
I20260620 20:22:52.692895 6366 ts_manager.cc:194] Registered new tserver with Master: b23046502b304dbc98a013e3192761b5 (127.31.235.130:39559)
I20260620 20:22:52.693761 6366 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:36989
WARNING: no policy specified for oryx/127.31.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.131.keytab.
Entry for principal oryx/127.31.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.131.keytab.
I20260620 20:22:52.725099 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33833
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.131.keytab
--principal=oryx/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:53.047701 6694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:53.047972 6694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:53.048064 6694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:53.055341 6694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:53.055456 6694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:53.055568 6694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:22:53.065167 6694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.131.keytab
--principal=oryx/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:53.066442 6694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:53.067636 6694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:53.080361 6700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:53.080300 6699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:53.080294 6702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:53.081419 6694 server_base.cc:1034] running on GCE node
I20260620 20:22:53.082095 6694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:53.083035 6694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:53.084282 6694 hybrid_clock.cc:648] HybridClock initialized: now 1781986973084218 us; error 48 us; skew 500 ppm
I20260620 20:22:53.084609 6694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:53 dist-test-slave-zzwf krb5kdc[6321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986973, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:53.087818 6694 init.cc:377] Logged in from keytab as oryx/127.31.235.131@KRBTEST.COM (short username oryx)
I20260620 20:22:53.089251 6694 webserver.cc:460] Webserver started at http://127.31.235.131:46277/ using document root <none> and password file <none>
I20260620 20:22:53.089991 6694 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:53.090149 6694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:53.090452 6694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:53.093237 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "e6c9422c263247ba8aae81e971905dc8"
format_stamp: "Formatted at 2026-06-20 20:22:53 on dist-test-slave-zzwf"
server_key: "f93b888329918af2c81b85d7bf7d2260"
server_key_iv: "d3fe348256aa7ad7d8705872507f20e1"
server_key_version: "encryptionkey@0"
I20260620 20:22:53.094126 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "e6c9422c263247ba8aae81e971905dc8"
format_stamp: "Formatted at 2026-06-20 20:22:53 on dist-test-slave-zzwf"
server_key: "f93b888329918af2c81b85d7bf7d2260"
server_key_iv: "d3fe348256aa7ad7d8705872507f20e1"
server_key_version: "encryptionkey@0"
I20260620 20:22:53.099118 6694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:22:53.102928 6709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:53.104348 6694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:22:53.104533 6694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "e6c9422c263247ba8aae81e971905dc8"
format_stamp: "Formatted at 2026-06-20 20:22:53 on dist-test-slave-zzwf"
server_key: "f93b888329918af2c81b85d7bf7d2260"
server_key_iv: "d3fe348256aa7ad7d8705872507f20e1"
server_key_version: "encryptionkey@0"
I20260620 20:22:53.104750 6694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:53.117395 6694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:53.121682 6694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:53.121973 6694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:53.122893 6694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:53.124183 6694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:53.124284 6694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:53.124434 6694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:53.124505 6694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:53.158879 6694 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:34255
I20260620 20:22:53.158898 6821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:34255 every 8 connection(s)
I20260620 20:22:53.160336 6694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:22:53.165371 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 6694
I20260620 20:22:53.165592 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:22:53.166024 32686 external_mini_cluster.cc:1428] Setting key d311a2a903bba0d8e231affd9557084a
Jun 20 20:22:53 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986973, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM
I20260620 20:22:53.180845 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:53.162884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:51763 (local address 127.31.235.190:33833)
0620 20:22:53.163103 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:53.163109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:53.164171 (+ 1062us) server_negotiation.cc:407] Connection header received
0620 20:22:53.166088 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:53.166094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:53.166188 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:53.166302 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:53.168004 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.168891 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.170438 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.170835 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.170968 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:53.174299 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:53.174304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:53.174307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:53.174338 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:53.177473 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:53.178234 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:53.178240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:53.178244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:53.178384 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:53.178877 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:53.178884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:53.178889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:53.179267 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:53.179408 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:53.180296 (+ 888us) server_negotiation.cc:299] Negotiation successful
0620 20:22:53.180506 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 20:22:53.182157 6824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:53.163345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.131:51763)
0620 20:22:53.164018 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:53.164053 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:53.165801 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:53.166561 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:53.166573 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:53.167147 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:53.167847 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:53.167866 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.169045 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.169049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:53.170295 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:53.170307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.171049 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.171058 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:53.171296 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:53.171952 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:53.171973 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:53.174090 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:53.177652 (+ 3562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:53.177659 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:53.177672 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:53.178085 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:53.178526 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:53.178530 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:53.178533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:53.178733 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:53.179410 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:53.179417 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:53.179801 (+ 384us) client_negotiation.cc:769] Sending connection context
0620 20:22:53.180179 (+ 378us) client_negotiation.cc:240] Negotiation successful
0620 20:22:53.180744 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":266,"threads_started":1}
I20260620 20:22:53.183501 6822 heartbeater.cc:347] Connected to a master server at 127.31.235.190:33833
I20260620 20:22:53.183828 6822 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:53.184505 6822 heartbeater.cc:511] Master 127.31.235.190:33833 requested a full tablet report, sending...
I20260620 20:22:53.185868 6366 ts_manager.cc:194] Registered new tserver with Master: e6c9422c263247ba8aae81e971905dc8 (127.31.235.131:34255)
I20260620 20:22:53.186658 6366 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:51763
I20260620 20:22:53.193714 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:22:53.209224 6555 heartbeater.cc:503] Master 127.31.235.190:33833 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.31.235.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.235.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.235.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.189.keytab.
Entry for principal oryx/127.31.235.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.189.keytab.
I20260620 20:22:53.241365 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.189:33151
--webserver_interface=127.31.235.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.189.keytab
--principal=oryx/127.31.235.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:33833,127.31.235.189:33151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:53.560956 6827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:53.561246 6827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:53.561334 6827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:53.568872 6827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:53.568974 6827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:53.569060 6827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:53.569166 6827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:53.569247 6827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:53.579347 6827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:33833,127.31.235.189:33151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.189.keytab
--principal=oryx/127.31.235.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.189:33151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.235.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
W20260620 20:22:53.579711 6827 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.31.235.190:33833,127.31.235.189:33151'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 20:22:53.580807 6827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:53.582172 6827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:53.593819 6832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:53.593791 6835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:53.593873 6833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:53.594487 6827 server_base.cc:1034] running on GCE node
I20260620 20:22:53.595263 6827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:53.596546 6827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:53.597754 6827 hybrid_clock.cc:648] HybridClock initialized: now 1781986973597707 us; error 27 us; skew 500 ppm
I20260620 20:22:53.598088 6827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:53 dist-test-slave-zzwf krb5kdc[6321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986973, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:53.600997 6827 init.cc:377] Logged in from keytab as oryx/127.31.235.189@KRBTEST.COM (short username oryx)
I20260620 20:22:53.602507 6827 webserver.cc:460] Webserver started at http://127.31.235.189:44575/ using document root <none> and password file <none>
I20260620 20:22:53.603114 6827 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:53.603204 6827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:53.603441 6827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:53.606257 6827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/data/instance:
uuid: "3799df284b594f518505074efea61789"
format_stamp: "Formatted at 2026-06-20 20:22:53 on dist-test-slave-zzwf"
server_key: "a8761fc8431fc88d3a1f291d20d27690"
server_key_iv: "51a25f794dc33690d72d26bcc8dc2736"
server_key_version: "encryptionkey@0"
I20260620 20:22:53.606940 6827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/wal/instance:
uuid: "3799df284b594f518505074efea61789"
format_stamp: "Formatted at 2026-06-20 20:22:53 on dist-test-slave-zzwf"
server_key: "a8761fc8431fc88d3a1f291d20d27690"
server_key_iv: "51a25f794dc33690d72d26bcc8dc2736"
server_key_version: "encryptionkey@0"
I20260620 20:22:53.612062 6827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:22:53.615429 6842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:53.616714 6827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 20:22:53.616884 6827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/wal
uuid: "3799df284b594f518505074efea61789"
format_stamp: "Formatted at 2026-06-20 20:22:53 on dist-test-slave-zzwf"
server_key: "a8761fc8431fc88d3a1f291d20d27690"
server_key_iv: "51a25f794dc33690d72d26bcc8dc2736"
server_key_version: "encryptionkey@0"
I20260620 20:22:53.617086 6827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:53.629189 6827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:53.633013 6827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:53.633286 6827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:53.651253 6827 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.189:33151
I20260620 20:22:53.651274 6893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.189:33151 every 8 connection(s)
I20260620 20:22:53.652609 6827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/data/info.pb
I20260620 20:22:53.656286 6894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:53.660724 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 6827
I20260620 20:22:53.663200 6894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:22:53 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.235.189@KRBTEST.COM
Jun 20 20:22:53 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986973, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.189@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM
I20260620 20:22:53.683566 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:53.668272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.31.235.190:33833)
0620 20:22:53.668475 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:53.668480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:53.669120 (+ 640us) server_negotiation.cc:407] Connection header received
0620 20:22:53.669384 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:53.669390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:53.669467 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:53.669585 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:53.671016 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.671721 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.673410 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.673934 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.674068 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:53.678202 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:53.678207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:53.678210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:53.678241 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:53.680674 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:53.681346 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:53.681352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:53.681356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:53.681543 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:53.681934 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:53.681942 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:53.681948 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:53.682271 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:53.682372 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:53.683013 (+ 641us) server_negotiation.cc:299] Negotiation successful
0620 20:22:53.683231 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:22:53.684659 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:53.662746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44546 (local address 127.31.235.189:33151)
0620 20:22:53.663548 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:53.663562 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:53.663615 (+ 53us) server_negotiation.cc:407] Connection header received
0620 20:22:53.666070 (+ 2455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:53.666100 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:53.666597 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:53.667022 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:53.669154 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.670580 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.671925 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.672596 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.672723 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:53.677642 (+ 4919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:53.677653 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:53.677668 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:53.677711 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:53.680691 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:53.681383 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:53.681389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:53.681391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:53.681591 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:53.681978 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:53.681981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:53.681983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:53.682562 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:53.682864 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:53.683182 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 20:22:53.684022 (+ 840us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":597,"server-negotiator.queue_time_us":556,"thread_start_us":329,"threads_started":1}
I20260620 20:22:53.684674 6898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:53.668253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.0.0.1:35280)
0620 20:22:53.668999 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:53.669043 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:53.669227 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:53.669774 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:53.669781 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:53.670508 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:53.670829 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:53.670844 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.671886 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.671891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:53.673277 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:53.673288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.674215 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.674223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:53.674437 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:53.675034 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:53.675067 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:53.678005 (+ 2938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:53.680809 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:53.680819 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:53.680835 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:53.681225 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:53.681652 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:53.681657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:53.681663 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:53.681827 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:53.682407 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:53.682414 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:53.682631 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 20:22:53.682932 (+ 301us) client_negotiation.cc:240] Negotiation successful
0620 20:22:53.683459 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":351,"threads_started":1}
I20260620 20:22:53.686587 6894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:22:53 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986973, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.189@KRBTEST.COM for oryx/127.31.235.189@KRBTEST.COM
I20260620 20:22:53.696355 6689 heartbeater.cc:503] Master 127.31.235.190:33833 was elected leader, sending a full tablet report...
I20260620 20:22:53.699509 6898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:53.687501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:33151 (local address 127.0.0.1:44556)
0620 20:22:53.687716 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:53.687732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:53.687818 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:53.688046 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:53.688049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:53.688346 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:53.688483 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:53.688490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.689423 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.689427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:53.690387 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:53.690397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.691126 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.691131 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:53.691258 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:53.691820 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:53.691836 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:53.693968 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:53.696962 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:53.696966 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:53.696970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:53.697401 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:53.697917 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:53.697921 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:53.697924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:53.698085 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:53.698586 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:53.698590 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:53.698724 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 20:22:53.698855 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 20:22:53.699134 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 20:22:53.699546 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:53.687532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44556 (local address 127.31.235.189:33151)
0620 20:22:53.687697 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:53.687701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:53.687789 (+ 88us) server_negotiation.cc:407] Connection header received
0620 20:22:53.687887 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:53.687890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:53.687941 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:53.688014 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:53.688591 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.689281 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.690540 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.690971 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.691087 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:53.694161 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:53.694165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:53.694169 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:53.694195 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:53.696804 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:53.697604 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:53.697608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:53.697610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:53.697747 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:53.698205 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:53.698212 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:53.698220 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:53.698479 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:53.698623 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:53.698922 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 20:22:53.699253 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":57}
I20260620 20:22:53.704200 6894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789: Bootstrap starting.
I20260620 20:22:53.707240 6894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:53.708381 6894 log.cc:824] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:53.711364 6894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789: No bootstrap required, opened a new log
I20260620 20:22:53.718442 6894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } }
I20260620 20:22:53.718788 6894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:53.718900 6894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3799df284b594f518505074efea61789, State: Initialized, Role: FOLLOWER
I20260620 20:22:53.719508 6894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } }
I20260620 20:22:53.720327 6900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } } }
I20260620 20:22:53.720860 6900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:22:53.721026 6901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:22:53.721279 6901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } }
I20260620 20:22:53.721763 6894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:53.722590 6901 leader_election.cc:288] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4efb3f7a9871466ca1c95f235aafdb86 (127.31.235.190:33833)
I20260620 20:22:53.723465 6376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3799df284b594f518505074efea61789" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4efb3f7a9871466ca1c95f235aafdb86" is_pre_election: true
I20260620 20:22:53.724642 6845 leader_election.cc:302] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 3799df284b594f518505074efea61789; no voters: 4efb3f7a9871466ca1c95f235aafdb86
I20260620 20:22:53.725114 6901 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260620 20:22:53.730786 6912 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:22:53.730924 6912 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:22:53.732322 6827 master_runner.cc:428] Detected that this master 3799df284b594f518505074efea61789 is joining an existing cluster
I20260620 20:22:53.732414 6827 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.235.189:33151
I20260620 20:22:53.753012 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:53.736365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35282 (local address 127.31.235.190:33833)
0620 20:22:53.736552 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:53.736559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:53.737328 (+ 769us) server_negotiation.cc:407] Connection header received
0620 20:22:53.737642 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:53.737649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:53.737728 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:53.737796 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:53.739353 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.740492 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.742543 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.743189 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.743288 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:53.745998 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:53.746004 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:53.746009 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:53.746047 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:53.749844 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:53.750721 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:53.750726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:53.750730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:53.750885 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:53.751321 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:53.751331 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:53.751334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:53.751677 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:53.751830 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:53.752408 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 20:22:53.752647 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:22:53.753273 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:53.736611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44572 (local address 127.31.235.189:33151)
0620 20:22:53.736796 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:53.736804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:53.737805 (+ 1001us) server_negotiation.cc:407] Connection header received
0620 20:22:53.738004 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:53.738009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:53.738071 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:53.738144 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:53.739967 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.741115 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.743207 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.744008 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.744112 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:53.746757 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:53.746765 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:53.746772 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:53.746807 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:53.750590 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:53.751289 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:53.751300 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:53.751304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:53.751448 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:53.751838 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:53.751845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:53.751851 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:53.752146 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:53.752296 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:53.752603 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 20:22:53.752921 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":57}
I20260620 20:22:53.753301 6919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:53.736515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:33151 (local address 127.0.0.1:44572)
0620 20:22:53.737710 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:53.737733 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:53.737884 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:53.738734 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:53.738739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:53.739464 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:53.739808 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:53.739822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.741369 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.741384 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:53.743061 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:53.743076 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.744412 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.744421 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:53.744680 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:53.745306 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:53.745331 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:53.746586 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:53.750718 (+ 4132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:53.750721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:53.750724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:53.751155 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:53.751568 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:53.751573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:53.751576 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:53.751723 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:53.752258 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:53.752262 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:53.752384 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 20:22:53.752517 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 20:22:53.752795 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"mutex_wait_us":73,"thread_start_us":780,"threads_started":1}
I20260620 20:22:53.754065 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:53.736501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.0.0.1:35282)
0620 20:22:53.737201 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:53.737226 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:53.737441 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:53.738017 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:53.738023 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:53.738735 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:53.739181 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:53.739196 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.740672 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.740677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:53.742362 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:53.742376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.743581 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.743587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:53.743758 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:53.744479 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:53.744510 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:53.745788 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:53.749976 (+ 4188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:53.749981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:53.749986 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:53.750576 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:53.751006 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:53.751011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:53.751017 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:53.751176 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:53.751808 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:53.751813 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:53.752026 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 20:22:53.753270 (+ 1244us) client_negotiation.cc:240] Negotiation successful
0620 20:22:53.753601 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":2,"spinlock_wait_cycles":27904,"thread_start_us":398,"threads_started":1}
WARNING: no policy specified for oryx/127.31.235.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.235.188@KRBTEST.COM" created.
Jun 20 20:22:53 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986971, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.190@KRBTEST.COM for oryx/127.31.235.189@KRBTEST.COM
I20260620 20:22:53.777334 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:53.759653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:33151 (local address 127.0.0.1:44584)
0620 20:22:53.760294 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:53.760332 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:53.760475 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:53.761243 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:53.761250 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:53.762061 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:53.762332 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:53.762344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.763808 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.763816 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:53.765515 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:53.765529 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.766359 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.766366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:53.766559 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:53.767212 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:53.767236 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:53.770112 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:53.774192 (+ 4080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:53.774207 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:53.774215 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:53.774788 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:53.775232 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:53.775237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:53.775240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:53.775397 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:53.776092 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:53.776101 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:53.776311 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 20:22:53.776524 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 20:22:53.776856 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"spinlock_wait_cycles":2432,"thread_start_us":394,"threads_started":1}
I20260620 20:22:53.778456 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:53.760011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.31.235.189:33151)
0620 20:22:53.760714 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:53.760720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:53.760738 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:22:53.760819 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:53.760824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:53.760921 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:53.761020 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:53.762514 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.763635 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.765658 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.766192 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.766289 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:53.770333 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:53.770339 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:53.770344 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:53.770381 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:53.774014 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:53.774929 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:53.774935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:53.774939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:53.775086 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:53.775522 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:53.775527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:53.775530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:53.775882 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:53.776018 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:53.777879 (+ 1861us) server_negotiation.cc:299] Negotiation successful
0620 20:22:53.778111 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591}
I20260620 20:22:53.780555 6366 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.235.189:33151: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3799df284b594f518505074efea61789"
member_type: NON_VOTER
last_known_addr {
host: "127.31.235.189"
port: 33151
}
attrs {
promote: true
}
}
dest_uuid: "4efb3f7a9871466ca1c95f235aafdb86"
cas_config_opid_index: -1
I20260620 20:22:53.781761 6366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true } }
I20260620 20:22:53.785818 6924 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3799df284b594f518505074efea61789 (127.31.235.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true } } }
I20260620 20:22:53.787343 6869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:53.790122 6924 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.235.189:33151
I20260620 20:22:53.790912 6827 master.cc:537] Master@127.31.235.189:33151 shutting down...
I20260620 20:22:53.790923 6925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4efb3f7a9871466ca1c95f235aafdb86" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true } } }
I20260620 20:22:53.791227 6925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:53.792443 6869 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 1 FOLLOWER]: Refusing update from remote peer 4efb3f7a9871466ca1c95f235aafdb86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 20:22:53.793419 6925 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.235.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.188.keytab.
Entry for principal oryx/127.31.235.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.188.keytab.
W20260620 20:22:53.796808 6353 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 20:22:53.797294 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.188:35317
--webserver_interface=127.31.235.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.188.keytab
--principal=oryx/127.31.235.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:33833,127.31.235.189:33151,127.31.235.188:35317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:53.802327 6353 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 -> Peer 3799df284b594f518505074efea61789 (127.31.235.189:33151): Couldn't send request to peer 3799df284b594f518505074efea61789. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 20:22:53.805161 6827 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:22:53.805598 6827 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:53.805786 6827 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789: stopping tablet replica
I20260620 20:22:53.822827 6827 master.cc:552] Master@127.31.235.189:33151 shutdown complete.
I20260620 20:22:53.822940 6827 master_runner.cc:305] Clearing existing system tablet
I20260620 20:22:53.823374 6827 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:53.823482 6827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:53.826443 6827 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 20:22:53.828001 6930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:53.828284 6827 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:53.828593 6827 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 20:22:53.828687 6827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/wal
uuid: "3799df284b594f518505074efea61789"
format_stamp: "Formatted at 2026-06-20 20:22:53 on dist-test-slave-zzwf"
server_key: "a8761fc8431fc88d3a1f291d20d27690"
server_key_iv: "51a25f794dc33690d72d26bcc8dc2736"
server_key_version: "encryptionkey@0"
I20260620 20:22:53.832125 6827 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:53.836913 6827 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 20:22:53.837108 6827 log.cc:1196] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 20:22:53.837554 6827 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789: Deleting consensus metadata
I20260620 20:22:53.837826 6827 master_runner.cc:315] Copying system tablet from 127.31.235.190:33833
I20260620 20:22:53.840188 6827 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789: tablet copy: Beginning tablet copy session from remote peer at address 127.31.235.190:33833
I20260620 20:22:53.852510 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:53.840944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.31.235.190:33833)
0620 20:22:53.841182 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:53.841192 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:53.841687 (+ 495us) server_negotiation.cc:407] Connection header received
0620 20:22:53.841911 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:53.841917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:53.842016 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:53.842129 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:53.842956 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.843701 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.845029 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.845432 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.845600 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:53.847344 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:53.847350 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:53.847354 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:53.847393 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:53.850014 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:53.850608 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:53.850615 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:53.850619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:53.850746 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:53.851102 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:53.851108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:53.851111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:53.851412 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:53.851571 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:53.851935 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 20:22:53.852140 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:22:53.852595 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:53.840929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.0.0.1:35294)
0620 20:22:53.841560 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:53.841589 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:53.841773 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:53.842157 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:53.842161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:53.842588 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:53.842812 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:53.842820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.843835 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.843841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:53.844878 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:53.844889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:53.845617 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:53.845622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:53.845817 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:53.846448 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:53.846465 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:53.847144 (+ 679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:53.850141 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:53.850149 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:53.850154 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:53.850488 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:53.850858 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:53.850863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:53.850867 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:53.850996 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:53.851529 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:53.851535 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:53.851695 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 20:22:53.851849 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 20:22:53.852140 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":397,"threads_started":1}
I20260620 20:22:53.853667 6386 tablet_copy_service.cc:140] P 4efb3f7a9871466ca1c95f235aafdb86: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3799df284b594f518505074efea61789 ({username='oryx', principal='oryx/127.31.235.189@KRBTEST.COM'} at 127.0.0.1:35294)
I20260620 20:22:53.853929 6386 tablet_copy_service.cc:161] P 4efb3f7a9871466ca1c95f235aafdb86: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3799df284b594f518505074efea61789 at {username='oryx', principal='oryx/127.31.235.189@KRBTEST.COM'} at 127.0.0.1:35294: session id = 3799df284b594f518505074efea61789-00000000000000000000000000000000
I20260620 20:22:53.856920 6386 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 20:22:53.858754 6827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 20:22:53.863376 6827 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789: tablet copy: Starting download of 0 data blocks...
I20260620 20:22:53.863701 6827 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789: tablet copy: Starting download of 1 WAL segments...
I20260620 20:22:53.866250 6827 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 20:22:53.869361 6386 tablet_copy_service.cc:342] P 4efb3f7a9871466ca1c95f235aafdb86: Request end of tablet copy session 3799df284b594f518505074efea61789-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.235.189@KRBTEST.COM'} at 127.0.0.1:35294
I20260620 20:22:53.869553 6386 tablet_copy_service.cc:433] P 4efb3f7a9871466ca1c95f235aafdb86: ending tablet copy session 3799df284b594f518505074efea61789-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3799df284b594f518505074efea61789
I20260620 20:22:53.879743 6827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:53.882406 6937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:53.888628 6938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:53.892460 6940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:53.897955 6827 server_base.cc:1034] running on GCE node
I20260620 20:22:53.898376 6827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:53.899050 6827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:53.900260 6827 hybrid_clock.cc:648] HybridClock initialized: now 1781986973900199 us; error 40 us; skew 500 ppm
I20260620 20:22:53.900490 6827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:53 dist-test-slave-zzwf krb5kdc[6321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986973, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:53.902379 6827 init.cc:377] Logged in from keytab as oryx/127.31.235.189@KRBTEST.COM (short username oryx)
I20260620 20:22:53.903656 6827 webserver.cc:460] Webserver started at http://127.31.235.189:33895/ using document root <none> and password file <none>
I20260620 20:22:53.903988 6827 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:53.904079 6827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:53.906446 6827 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 20:22:53.908078 6947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:53.908501 6827 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 20:22:53.908627 6827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/wal
uuid: "3799df284b594f518505074efea61789"
format_stamp: "Formatted at 2026-06-20 20:22:53 on dist-test-slave-zzwf"
server_key: "a8761fc8431fc88d3a1f291d20d27690"
server_key_iv: "51a25f794dc33690d72d26bcc8dc2736"
server_key_version: "encryptionkey@0"
I20260620 20:22:53.908804 6827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:53.938117 6827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:53.941547 6827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:53.961216 6998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.189:33151 every 8 connection(s)
I20260620 20:22:53.961331 6827 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.189:33151
I20260620 20:22:53.962193 6827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-1/data/info.pb
I20260620 20:22:53.969199 6999 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:22:53.971676 6999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789: Bootstrap starting.
I20260620 20:22:53.983868 6999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:22:53.984401 6999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789: Bootstrap complete.
I20260620 20:22:53.985294 6999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true } }
I20260620 20:22:53.985550 6999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3799df284b594f518505074efea61789, State: Initialized, Role: LEARNER
I20260620 20:22:53.985771 6999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true } }
I20260620 20:22:53.986552 7002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true } } }
I20260620 20:22:53.986877 6999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:53.987051 7002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [sys.catalog]: This master's current role is: LEARNER
I20260620 20:22:53.998077 7013 catalog_manager.cc:1259] Loaded cluster ID: 66a41b8acf634ce197e6bc9780a4c844
I20260620 20:22:53.998211 7013 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789: loading cluster ID for follower catalog manager: success
I20260620 20:22:54.000521 7013 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789: acquiring CA information for follower catalog manager: success
I20260620 20:22:54.002369 7013 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:22:54.189077 6822 heartbeater.cc:503] Master 127.31.235.190:33833 was elected leader, sending a full tablet report...
W20260620 20:22:54.199651 6928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:54.199915 6928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:54.200057 6928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:54.207329 6928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:22:54.207432 6928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:54.207515 6928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:54.207623 6928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:22:54.207712 6928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:22:54.219046 6928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:33833,127.31.235.189:33151,127.31.235.188:35317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.188.keytab
--principal=oryx/127.31.235.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.188:35317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.235.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:54.220309 6928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:54.221700 6928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:54.233287 7018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:54.233289 7019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:54.233340 7021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:54.234143 6928 server_base.cc:1034] running on GCE node
I20260620 20:22:54.234802 6928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:54.236140 6928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:54.237329 6928 hybrid_clock.cc:648] HybridClock initialized: now 1781986974237275 us; error 61 us; skew 500 ppm
I20260620 20:22:54.237679 6928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:54 dist-test-slave-zzwf krb5kdc[6321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986974, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:54.240680 6928 init.cc:377] Logged in from keytab as oryx/127.31.235.188@KRBTEST.COM (short username oryx)
I20260620 20:22:54.242260 6928 webserver.cc:460] Webserver started at http://127.31.235.188:34603/ using document root <none> and password file <none>
I20260620 20:22:54.242920 6928 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:54.243064 6928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:54.243361 6928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:22:54.246253 6928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/data/instance:
uuid: "faaa7365b68646198e123e92c3e81b12"
format_stamp: "Formatted at 2026-06-20 20:22:54 on dist-test-slave-zzwf"
server_key: "4c49a54608d278f82b37caaaeb3042b8"
server_key_iv: "4d68d789ea62b046c8571a0a803a991c"
server_key_version: "encryptionkey@0"
I20260620 20:22:54.246979 6928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/wal/instance:
uuid: "faaa7365b68646198e123e92c3e81b12"
format_stamp: "Formatted at 2026-06-20 20:22:54 on dist-test-slave-zzwf"
server_key: "4c49a54608d278f82b37caaaeb3042b8"
server_key_iv: "4d68d789ea62b046c8571a0a803a991c"
server_key_version: "encryptionkey@0"
I20260620 20:22:54.259936 6928 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.008s sys 0.004s
I20260620 20:22:54.263460 7028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:54.264707 6928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:22:54.264881 6928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/wal
uuid: "faaa7365b68646198e123e92c3e81b12"
format_stamp: "Formatted at 2026-06-20 20:22:54 on dist-test-slave-zzwf"
server_key: "4c49a54608d278f82b37caaaeb3042b8"
server_key_iv: "4d68d789ea62b046c8571a0a803a991c"
server_key_version: "encryptionkey@0"
I20260620 20:22:54.265065 6928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:54.283558 6928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:54.288341 6928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:54.288627 6928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:54.306180 6928 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.188:35317
I20260620 20:22:54.306200 7079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.188:35317 every 8 connection(s)
I20260620 20:22:54.307613 6928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/data/info.pb
I20260620 20:22:54.308725 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 6928
I20260620 20:22:54.312449 7080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:54.322587 7080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:22:54 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.235.188@KRBTEST.COM
I20260620 20:22:54.333257 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:54.311082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41880 (local address 127.31.235.188:35317)
0620 20:22:54.311845 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:54.311856 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:54.311895 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:22:54.313480 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:54.313574 (+ 94us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:54.313994 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:54.314351 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:54.316165 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.317616 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.319625 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.320295 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.320482 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:54.325219 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:54.325229 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:54.325247 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:54.325290 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:54.329062 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:54.329911 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:54.329917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:54.329922 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:54.330103 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:54.330494 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:54.330497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:54.330499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:54.331053 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:54.331334 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:54.331774 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 20:22:54.332203 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":344,"threads_started":1}
Jun 20 20:22:54 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986974, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.188@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM
I20260620 20:22:54.340085 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:54.325564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.0.0.1:35304)
0620 20:22:54.326394 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:54.326434 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:54.326605 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:54.326981 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:54.326986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:54.327541 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:54.327795 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:54.327803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.329211 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.329215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:54.330772 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:54.330783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.331715 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.331726 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:54.331928 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:54.332548 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:54.332566 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:54.335079 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:54.338072 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:54.338080 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:54.338085 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:54.338475 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:54.338800 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:54.338803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:54.338805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:54.338918 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:54.339360 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:54.339365 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:54.339491 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 20:22:54.339627 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 20:22:54.339823 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":437,"threads_started":1}
I20260620 20:22:54.340092 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:54.325567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35304 (local address 127.31.235.190:33833)
0620 20:22:54.325782 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:54.325790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:54.326561 (+ 771us) server_negotiation.cc:407] Connection header received
0620 20:22:54.326719 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:54.326726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:54.326806 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:54.326927 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:54.327944 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.329044 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.330899 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.331505 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.331634 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:54.335255 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:54.335260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:54.335263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:54.335293 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:54.337921 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:54.338585 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:54.338589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:54.338591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:54.338709 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:54.339017 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:54.339020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:54.339022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:54.339263 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:54.339371 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:54.339714 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 20:22:54.339873 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:22:54.341293 7080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:22:54 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986974, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.188@KRBTEST.COM for oryx/127.31.235.189@KRBTEST.COM
I20260620 20:22:54.353210 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:54.341918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:33151 (local address 127.0.0.1:44596)
0620 20:22:54.342117 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:54.342134 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:54.342222 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:54.342830 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:54.342833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:54.343120 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:54.343235 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:54.343242 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.344466 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.344469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:54.345360 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:54.345370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.346095 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.346100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:54.346225 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:54.346698 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:54.346714 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:54.348568 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:54.351420 (+ 2852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:54.351424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:54.351426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:54.351769 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:54.352074 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:54.352077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:54.352079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:54.352193 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:54.352593 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:54.352596 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:54.352695 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 20:22:54.352796 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 20:22:54.352983 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 20:22:54.353308 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:54.342065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44596 (local address 127.31.235.189:33151)
0620 20:22:54.342595 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:54.342599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:54.342614 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:22:54.342659 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:54.342662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:54.342714 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:54.342797 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:54.343368 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.344328 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.345479 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.345962 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.346053 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:54.348734 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:54.348738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:54.348741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:54.348771 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:54.351307 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:54.351862 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:54.351866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:54.351869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:54.351985 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:54.352278 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:54.352281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:54.352283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:54.352506 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:54.352614 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:54.352878 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 20:22:54.353017 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":309,"threads_started":1}
I20260620 20:22:54.354333 7080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:22:54 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986974, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.188@KRBTEST.COM for oryx/127.31.235.188@KRBTEST.COM
I20260620 20:22:54.366048 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:54.354938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:35317 (local address 127.0.0.1:41894)
0620 20:22:54.355087 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:54.355102 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:54.355172 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:54.355489 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:54.355493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:54.355815 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:54.355945 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:54.355952 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.356863 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.356867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:54.357815 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:54.357826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.358439 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.358443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:54.358563 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:54.359064 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:54.359080 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:54.360953 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:54.363694 (+ 2741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:54.363698 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:54.363700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:54.364108 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:54.364404 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:54.364407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:54.364409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:54.364543 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:54.365007 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:54.365010 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:54.365132 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 20:22:54.365287 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 20:22:54.365571 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 20:22:54.366079 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:54.355095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41894 (local address 127.31.235.188:35317)
0620 20:22:54.355232 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:54.355236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:54.355249 (+ 13us) server_negotiation.cc:407] Connection header received
0620 20:22:54.355284 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:54.355287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:54.355334 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:54.355408 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:54.356067 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.356729 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.357939 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.358309 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.358395 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:54.361108 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:54.361112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:54.361115 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:54.361140 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:54.363570 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:54.364193 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:54.364197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:54.364199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:54.364315 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:54.364644 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:54.364647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:54.364649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:54.364878 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:54.364979 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:54.365366 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 20:22:54.365720 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":44}
I20260620 20:22:54.370643 7080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: Bootstrap starting.
I20260620 20:22:54.373624 7080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:54.374621 7080 log.cc:824] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:54.377053 7080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: No bootstrap required, opened a new log
I20260620 20:22:54.383921 7080 raft_consensus.cc:348] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } }
I20260620 20:22:54.384213 7080 raft_consensus.cc:374] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:54.384351 7080 raft_consensus.cc:729] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faaa7365b68646198e123e92c3e81b12, State: Initialized, Role: FOLLOWER
I20260620 20:22:54.384886 7080 consensus_queue.cc:260] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } }
I20260620 20:22:54.385855 7087 sys_catalog.cc:455] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } } }
I20260620 20:22:54.386210 7087 sys_catalog.cc:458] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:22:54.387072 7080 sys_catalog.cc:564] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 20:22:54.396023 7098 catalog_manager.cc:1558] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:22:54.396180 7098 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:22:54.400218 6928 master_runner.cc:428] Detected that this master faaa7365b68646198e123e92c3e81b12 is joining an existing cluster
I20260620 20:22:54.400321 6928 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.235.188:35317
I20260620 20:22:54.429601 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:54.404183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.0.0.1:35314)
0620 20:22:54.405280 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:54.405301 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:54.405471 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:54.406052 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:54.406060 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:54.406933 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:54.407436 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:54.407452 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.409573 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.409581 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:54.412181 (+ 2600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:54.412199 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.413763 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.413769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:54.414013 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:54.414884 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:54.414910 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:54.416138 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:54.426288 (+ 10150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:54.426295 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:54.426302 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:54.426911 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:54.427433 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:54.427439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:54.427445 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:54.427642 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:54.428315 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:54.428320 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:54.428549 (+ 229us) client_negotiation.cc:769] Sending connection context
0620 20:22:54.428713 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 20:22:54.429065 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"thread_start_us":806,"threads_started":1}
I20260620 20:22:54.430619 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:54.404600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35314 (local address 127.31.235.190:33833)
0620 20:22:54.404819 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:54.404827 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:54.405370 (+ 543us) server_negotiation.cc:407] Connection header received
0620 20:22:54.405662 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:54.405669 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:54.405750 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:54.405824 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:54.407636 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.409176 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.412386 (+ 3210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.413249 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.413371 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:54.416329 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:54.416336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:54.416342 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:54.416388 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:54.419940 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:54.427031 (+ 7091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:54.427039 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:54.427046 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:54.427215 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:54.427775 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:54.427780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:54.427783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:54.428118 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:54.428248 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:54.429930 (+ 1682us) server_negotiation.cc:299] Negotiation successful
0620 20:22:54.430241 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:22:54.435026 7108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:54.420240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:33151 (local address 127.0.0.1:44612)
0620 20:22:54.427689 (+ 7449us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:54.427708 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:54.427813 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:54.428396 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:54.428400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:54.428425 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:54.428658 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:54.428672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.430086 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.430093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:54.432085 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:54.432098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.434209 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.434217 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:54.434395 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:54.434449 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 20:22:54.434555 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 20:22:54.434694 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7338,"thread_start_us":10505,"threads_started":1}
I20260620 20:22:54.435660 7109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:54.420654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44612 (local address 127.31.235.189:33151)
0620 20:22:54.427349 (+ 6695us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:54.427355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:54.427917 (+ 562us) server_negotiation.cc:407] Connection header received
0620 20:22:54.427975 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:54.427979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:54.428059 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:54.428173 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:54.428819 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.429942 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.432247 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.433645 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.433893 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:54.434095 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:54.435202 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0620 20:22:54.435332 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6552,"thread_start_us":6646,"threads_started":1}
I20260620 20:22:54.436908 6974 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 20:22:54.437513 7002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4efb3f7a9871466ca1c95f235aafdb86. Latest consensus state: current_term: 1 leader_uuid: "4efb3f7a9871466ca1c95f235aafdb86" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true } } }
I20260620 20:22:54.437774 7002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [sys.catalog]: This master's current role is: LEARNER
I20260620 20:22:54.440761 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 6427
Jun 20 20:22:54 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986971, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.190@KRBTEST.COM for oryx/127.31.235.188@KRBTEST.COM
I20260620 20:22:54.449354 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:54.405747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41898 (local address 127.31.235.188:35317)
0620 20:22:54.405917 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:54.405923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:54.425422 (+ 19499us) server_negotiation.cc:407] Connection header received
0620 20:22:54.425614 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:54.425622 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:54.425700 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:54.425780 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:54.428449 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.429584 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.439273 (+ 9689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.440045 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.440169 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:54.442488 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:54.442494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:54.442502 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:54.442536 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:54.446217 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:54.447027 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:54.447036 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:54.447041 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:54.447162 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:54.447524 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:54.447529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:54.447532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:54.447867 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:54.448061 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:54.448756 (+ 695us) server_negotiation.cc:299] Negotiation successful
0620 20:22:54.449026 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":56}
I20260620 20:22:54.449391 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:54.405437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:35317 (local address 127.0.0.1:41898)
0620 20:22:54.425289 (+ 19852us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:54.425312 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:54.425447 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:54.425987 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:54.425993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:54.427946 (+ 1953us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:54.428294 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:54.428310 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.430125 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.430130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:54.439099 (+ 8969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:54.439119 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.440236 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.440245 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:54.440445 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:54.441107 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:54.441133 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:54.442318 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:54.446357 (+ 4039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:54.446364 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:54.446371 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:54.446888 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:54.447280 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:54.447285 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:54.447288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:54.447412 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:54.448111 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:54.448115 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:54.448402 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 20:22:54.448520 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 20:22:54.448874 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19681,"mutex_wait_us":93,"spinlock_wait_cycles":30592,"thread_start_us":10117,"threads_started":1}
I20260620 20:22:54.450294 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:54.404130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44604 (local address 127.31.235.189:33151)
0620 20:22:54.404311 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:54.404320 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:54.415530 (+ 11210us) server_negotiation.cc:407] Connection header received
0620 20:22:54.415921 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:54.415928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:54.416022 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:54.416096 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:54.419284 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.420329 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.422765 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.423636 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.423748 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:54.431016 (+ 7268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:54.431023 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:54.431028 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:54.431067 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:54.434679 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:54.436077 (+ 1398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:54.436083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:54.436086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:54.436273 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:54.438435 (+ 2162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:54.438442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:54.438447 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:54.438814 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:54.438965 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:54.449712 (+ 10747us) server_negotiation.cc:299] Negotiation successful
0620 20:22:54.449961 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:22:54.451042 7107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:54.404262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:33151 (local address 127.0.0.1:44604)
0620 20:22:54.415419 (+ 11157us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:54.415437 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:54.415766 (+ 329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:54.416300 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:54.416305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:54.416991 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:54.417360 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:54.417373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.420546 (+ 3173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.420556 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:54.422581 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:54.422599 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.424062 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.424070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:54.424326 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:54.425807 (+ 1481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:54.425838 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:54.427615 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:54.434844 (+ 7229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:54.434849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:54.434853 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:54.435551 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:54.436450 (+ 899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:54.436454 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:54.436457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:54.438309 (+ 1852us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:54.439026 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:54.439031 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:54.448193 (+ 9162us) client_negotiation.cc:769] Sending connection context
0620 20:22:54.448366 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 20:22:54.450584 (+ 2218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11001,"thread_start_us":20663,"threads_started":1}
I20260620 20:22:54.456297 7108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:54.435864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:35317 (local address 127.0.0.1:41906)
0620 20:22:54.436598 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:54.436620 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:54.436694 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:54.440822 (+ 4128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:54.440828 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:54.441315 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:54.441540 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:54.441553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.443399 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.443408 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:54.444843 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:54.444859 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.445976 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.445982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:54.446167 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:54.446755 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:54.446780 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:54.449381 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:54.453809 (+ 4428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:54.453815 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:54.453819 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:54.454195 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:54.454594 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:54.454598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:54.454601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:54.454739 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:54.455378 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:54.455385 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:54.455547 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 20:22:54.455695 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 20:22:54.455970 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600}
I20260620 20:22:54.456422 7110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:54.435907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41906 (local address 127.31.235.188:35317)
0620 20:22:54.440377 (+ 4470us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:54.440383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:54.440435 (+ 52us) server_negotiation.cc:407] Connection header received
0620 20:22:54.440537 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:54.440543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:54.440641 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:54.441686 (+ 1045us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:54.442009 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.443245 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.445026 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.445792 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.448108 (+ 2316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:54.449794 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:54.449800 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:54.449805 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:54.449833 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:54.453660 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:54.454328 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:54.454333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:54.454336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:54.454465 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:54.454863 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:54.454869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:54.454872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:54.455229 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:54.455362 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:54.455822 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 20:22:54.456017 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1415,"server-negotiator.queue_time_us":3759,"thread_start_us":3615,"threads_started":1}
I20260620 20:22:54.457787 6366 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.235.188:35317: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "faaa7365b68646198e123e92c3e81b12"
member_type: NON_VOTER
last_known_addr {
host: "127.31.235.188"
port: 35317
}
attrs {
promote: true
}
}
dest_uuid: "4efb3f7a9871466ca1c95f235aafdb86"
cas_config_opid_index: 5
I20260620 20:22:54.458814 6366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: NON_VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: true } }
I20260620 20:22:54.460634 7106 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER faaa7365b68646198e123e92c3e81b12 (127.31.235.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: NON_VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: true } } }
I20260620 20:22:54.462461 6974 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 1 LEARNER]: Refusing update from remote peer 4efb3f7a9871466ca1c95f235aafdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
Jun 20 20:22:54 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.235.129@KRBTEST.COM
I20260620 20:22:54.463732 7055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:54.467890 7055 raft_consensus.cc:1261] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 1 FOLLOWER]: Refusing update from remote peer 4efb3f7a9871466ca1c95f235aafdb86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 20:22:54.470871 7114 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 20:22:54.471053 7087 raft_consensus.cc:482] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:22:54.471376 7087 raft_consensus.cc:504] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } }
I20260620 20:22:54.472921 7087 leader_election.cc:288] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4efb3f7a9871466ca1c95f235aafdb86 (127.31.235.190:33833), 3799df284b594f518505074efea61789 (127.31.235.189:33151)
I20260620 20:22:54.473636 6376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "faaa7365b68646198e123e92c3e81b12" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4efb3f7a9871466ca1c95f235aafdb86" is_pre_election: true
I20260620 20:22:54.473626 6974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "faaa7365b68646198e123e92c3e81b12" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3799df284b594f518505074efea61789" is_pre_election: true
I20260620 20:22:54.474501 7032 leader_election.cc:302] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: faaa7365b68646198e123e92c3e81b12; no voters: 3799df284b594f518505074efea61789, 4efb3f7a9871466ca1c95f235aafdb86
I20260620 20:22:54.474943 7087 raft_consensus.cc:2726] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:22:54.476733 7114 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: NON_VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:54.480468 7111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:54.441889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48552 (local address 127.31.235.129:37795)
0620 20:22:54.452457 (+ 10568us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:54.452469 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:54.452513 (+ 44us) server_negotiation.cc:407] Connection header received
0620 20:22:54.452582 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:54.452591 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:54.452890 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:54.453086 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:54.454817 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.456178 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.457966 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.458596 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.458734 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:54.472315 (+ 13581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:54.472336 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:54.472356 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:54.472414 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:54.476137 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:54.477124 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:54.477139 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:54.477145 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:54.477340 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:54.477874 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:54.477878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:54.477881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:54.478340 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:54.479603 (+ 1263us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:54.479761 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 20:22:54.480043 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10365,"thread_start_us":10408,"threads_started":1}
I20260620 20:22:54.481848 6550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:54.490123 6974 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER faaa7365b68646198e123e92c3e81b12 (127.31.235.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: NON_VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: true } } }
I20260620 20:22:54.492794 7055 raft_consensus.cc:858] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: NON_VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: true } }
W20260620 20:22:54.493185 7055 raft_consensus.cc:1565] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: NON_VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: true } }
I20260620 20:22:54.495476 7002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4efb3f7a9871466ca1c95f235aafdb86" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: NON_VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: true } } }
I20260620 20:22:54.495833 7002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [sys.catalog]: This master's current role is: LEARNER
I20260620 20:22:54.504774 7106 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.235.188:35317
I20260620 20:22:54.505731 7106 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86: attempting to promote NON_VOTER 3799df284b594f518505074efea61789 to VOTER
I20260620 20:22:54.505659 7112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4efb3f7a9871466ca1c95f235aafdb86" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: NON_VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: true } } }
I20260620 20:22:54.506129 7112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:54.506213 7055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3799df284b594f518505074efea61789 (127.31.235.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true } } }
I20260620 20:22:54.506645 7106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: NON_VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: true } }
I20260620 20:22:54.508253 7114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4efb3f7a9871466ca1c95f235aafdb86" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: NON_VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: NON_VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: true } } }
I20260620 20:22:54.508754 7114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:54.511322 6974 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 1 LEARNER]: Refusing update from remote peer 4efb3f7a9871466ca1c95f235aafdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 20:22:54.512145 7114 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 20:22:54.515072 7113 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3799df284b594f518505074efea61789 (127.31.235.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: NON_VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: true } } }
I20260620 20:22:54.515825 6974 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3799df284b594f518505074efea61789 (127.31.235.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: NON_VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: true } } }
I20260620 20:22:54.519003 7113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4efb3f7a9871466ca1c95f235aafdb86" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: NON_VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: true } } }
I20260620 20:22:54.519568 7113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:54.519614 7002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4efb3f7a9871466ca1c95f235aafdb86" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: NON_VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: true } } }
I20260620 20:22:54.519858 7002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:22:54.519263 7112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4efb3f7a9871466ca1c95f235aafdb86" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: NON_VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: true } } }
I20260620 20:22:54.520120 7112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:54.524207 6928 master.cc:537] Master@127.31.235.188:35317 shutting down...
I20260620 20:22:54.540159 7054 raft_consensus.cc:1467] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20260620 20:22:54.540424 7054 raft_consensus.cc:1261] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 1 NON_PARTICIPANT]: Refusing update from remote peer 4efb3f7a9871466ca1c95f235aafdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 20:22:54.540506 7087 sys_catalog.cc:455] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4efb3f7a9871466ca1c95f235aafdb86. Latest consensus state: current_term: 1 leader_uuid: "4efb3f7a9871466ca1c95f235aafdb86" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true } } }
I20260620 20:22:54.540771 7087 sys_catalog.cc:458] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 20:22:54.541666 7114 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: NON_VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 20:22:54.541211 7119 sys_catalog.cc:455] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4efb3f7a9871466ca1c95f235aafdb86" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true } } }
I20260620 20:22:54.542037 7119 sys_catalog.cc:458] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [sys.catalog]: This master's current role is: NON_PARTICIPANT
W20260620 20:22:54.544692 6352 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 -> Peer faaa7365b68646198e123e92c3e81b12 (127.31.235.188:35317): Couldn't send request to peer faaa7365b68646198e123e92c3e81b12. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 20:22:54.545257 6928 raft_consensus.cc:2229] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260620 20:22:54.545737 6928 raft_consensus.cc:2258] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260620 20:22:54.545010 7124 sys_catalog.cc:455] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4efb3f7a9871466ca1c95f235aafdb86" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: NON_VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: true } } }
I20260620 20:22:54.545969 7124 sys_catalog.cc:458] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 20:22:54.546180 6928 tablet_replica.cc:330] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: stopping tablet replica
I20260620 20:22:54.553905 6928 master.cc:552] Master@127.31.235.188:35317 shutdown complete.
I20260620 20:22:54.554014 6928 master_runner.cc:305] Clearing existing system tablet
I20260620 20:22:54.554399 6928 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:54.554502 6928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:54.557343 6928 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 20:22:54.559023 7127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:54.559365 6928 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:54.559620 6928 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 20:22:54.559718 6928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/wal
uuid: "faaa7365b68646198e123e92c3e81b12"
format_stamp: "Formatted at 2026-06-20 20:22:54 on dist-test-slave-zzwf"
server_key: "4c49a54608d278f82b37caaaeb3042b8"
server_key_iv: "4d68d789ea62b046c8571a0a803a991c"
server_key_version: "encryptionkey@0"
I20260620 20:22:54.564395 6928 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:54.571168 6928 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 20:22:54.571333 6928 log.cc:1196] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 20:22:54.571849 6928 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: Deleting consensus metadata
I20260620 20:22:54.572106 6928 master_runner.cc:315] Copying system tablet from 127.31.235.190:33833
I20260620 20:22:54.593133 6928 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: tablet copy: Beginning tablet copy session from remote peer at address 127.31.235.190:33833
I20260620 20:22:54.612339 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:54.594005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.0.0.1:35326)
0620 20:22:54.597906 (+ 3901us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:54.597932 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:54.598115 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:54.598501 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:54.598548 (+ 47us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:54.599075 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:54.599338 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:54.599349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.600849 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.600857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:54.602467 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:54.602482 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.603381 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.603387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:54.603565 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:54.604273 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:54.604296 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:54.605350 (+ 1054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:54.609229 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:54.609237 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:54.609245 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:54.609758 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:54.610289 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:54.610294 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:54.610297 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:54.610484 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:54.611181 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:54.611187 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:54.611379 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 20:22:54.611601 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 20:22:54.611918 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3745,"thread_start_us":3645,"threads_started":1}
I20260620 20:22:54.612362 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:54.594055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35326 (local address 127.31.235.190:33833)
0620 20:22:54.594250 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:54.594258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:54.598066 (+ 3808us) server_negotiation.cc:407] Connection header received
0620 20:22:54.598251 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:54.598258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:54.598342 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:54.598452 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:54.599503 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.600692 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.602620 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:54.603216 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:54.603337 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:54.605569 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:54.605576 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:54.605581 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:54.605618 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:54.609086 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:54.610019 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:54.610024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:54.610028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:54.610170 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:54.610667 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:54.610672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:54.610676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:54.611054 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:54.611209 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:54.611730 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 20:22:54.611996 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:22:54.613348 6386 tablet_copy_service.cc:140] P 4efb3f7a9871466ca1c95f235aafdb86: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer faaa7365b68646198e123e92c3e81b12 ({username='oryx', principal='oryx/127.31.235.188@KRBTEST.COM'} at 127.0.0.1:35326)
I20260620 20:22:54.613538 6386 tablet_copy_service.cc:161] P 4efb3f7a9871466ca1c95f235aafdb86: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer faaa7365b68646198e123e92c3e81b12 at {username='oryx', principal='oryx/127.31.235.188@KRBTEST.COM'} at 127.0.0.1:35326: session id = faaa7365b68646198e123e92c3e81b12-00000000000000000000000000000000
I20260620 20:22:54.617075 6386 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 20:22:54.619033 6928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 20:22:54.625620 6928 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: tablet copy: Starting download of 0 data blocks...
I20260620 20:22:54.625962 6928 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: tablet copy: Starting download of 1 WAL segments...
I20260620 20:22:54.628192 6928 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 20:22:54.632383 6386 tablet_copy_service.cc:342] P 4efb3f7a9871466ca1c95f235aafdb86: Request end of tablet copy session faaa7365b68646198e123e92c3e81b12-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.235.188@KRBTEST.COM'} at 127.0.0.1:35326
I20260620 20:22:54.632510 6386 tablet_copy_service.cc:433] P 4efb3f7a9871466ca1c95f235aafdb86: ending tablet copy session faaa7365b68646198e123e92c3e81b12-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer faaa7365b68646198e123e92c3e81b12
I20260620 20:22:54.635788 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 6427
I20260620 20:22:54.649271 6928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:54.652099 7134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:54.654755 7135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:54.659121 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:37795
--local_ip_for_outbound_sockets=127.31.235.129
--tserver_master_addrs=127.31.235.190:33833
--webserver_port=42717
--webserver_interface=127.31.235.129
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.129.keytab
--principal=oryx/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.235.190:33833,127.31.235.189:33151
--tserver_master_addrs=127.31.235.190:33833,127.31.235.189:33151,127.31.235.188:35317 with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:54.661079 7137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:54.668215 6928 server_base.cc:1034] running on GCE node
I20260620 20:22:54.668551 6928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:54.669190 6928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:54.670413 6928 hybrid_clock.cc:648] HybridClock initialized: now 1781986974670353 us; error 41 us; skew 500 ppm
I20260620 20:22:54.670677 6928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:54 dist-test-slave-zzwf krb5kdc[6321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986974, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:54.672655 6928 init.cc:377] Logged in from keytab as oryx/127.31.235.188@KRBTEST.COM (short username oryx)
I20260620 20:22:54.677738 6928 webserver.cc:460] Webserver started at http://127.31.235.188:44481/ using document root <none> and password file <none>
I20260620 20:22:54.678057 6928 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:54.678176 6928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:54.680538 6928 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 20:22:54.682260 7145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:54.682693 6928 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 20:22:54.682829 6928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/wal
uuid: "faaa7365b68646198e123e92c3e81b12"
format_stamp: "Formatted at 2026-06-20 20:22:54 on dist-test-slave-zzwf"
server_key: "4c49a54608d278f82b37caaaeb3042b8"
server_key_iv: "4d68d789ea62b046c8571a0a803a991c"
server_key_version: "encryptionkey@0"
I20260620 20:22:54.683018 6928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:54.693897 6928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:54.697227 6928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:54.728390 6928 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.188:35317
I20260620 20:22:54.728408 7196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.188:35317 every 8 connection(s)
I20260620 20:22:54.729295 6928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/master-2/data/info.pb
I20260620 20:22:54.736330 7197 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:22:54.741848 7197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: Bootstrap starting.
I20260620 20:22:54.753918 7197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:22:54.754359 7197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: Bootstrap complete.
I20260620 20:22:54.755203 7197 raft_consensus.cc:348] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: NON_VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: true } }
I20260620 20:22:54.755465 7197 raft_consensus.cc:729] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: faaa7365b68646198e123e92c3e81b12, State: Initialized, Role: LEARNER
I20260620 20:22:54.755688 7197 consensus_queue.cc:260] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: NON_VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: true } }
I20260620 20:22:54.758061 7200 sys_catalog.cc:455] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: NON_VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: true } } }
I20260620 20:22:54.758299 7197 sys_catalog.cc:564] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:22:54.758468 7200 sys_catalog.cc:458] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [sys.catalog]: This master's current role is: LEARNER
I20260620 20:22:54.776549 7211 catalog_manager.cc:1259] Loaded cluster ID: 66a41b8acf634ce197e6bc9780a4c844
I20260620 20:22:54.776679 7211 catalog_manager.cc:1552] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: loading cluster ID for follower catalog manager: success
I20260620 20:22:54.779196 7211 catalog_manager.cc:1574] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: acquiring CA information for follower catalog manager: success
I20260620 20:22:54.780845 7211 catalog_manager.cc:1602] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 20:22:55.052591 7139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:55.052906 7139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:55.053059 7139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:55.060663 7139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:55.060820 7139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:55.060989 7139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:22:55.071251 7139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.129.keytab
--principal=oryx/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:37795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=42717
--tserver_master_addrs=127.31.235.190:33833,127.31.235.189:33151,127.31.235.188:35317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:55.072613 7139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:55.074311 7139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:55.088052 7216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:55.088018 7217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:55.088254 7219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:55.089335 7139 server_base.cc:1034] running on GCE node
I20260620 20:22:55.090024 7139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:55.090960 7139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:55.092150 7139 hybrid_clock.cc:648] HybridClock initialized: now 1781986975092085 us; error 61 us; skew 500 ppm
I20260620 20:22:55.092456 7139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:55 dist-test-slave-zzwf krb5kdc[6321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986975, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:55.095487 7139 init.cc:377] Logged in from keytab as oryx/127.31.235.129@KRBTEST.COM (short username oryx)
I20260620 20:22:55.096839 7139 webserver.cc:460] Webserver started at http://127.31.235.129:42717/ using document root <none> and password file <none>
I20260620 20:22:55.097546 7139 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:55.097697 7139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:55.102866 7139 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260620 20:22:55.105481 7226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:55.106762 7139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:22:55.106920 7139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "838a74503a5f46b39c7553da21104e3d"
format_stamp: "Formatted at 2026-06-20 20:22:52 on dist-test-slave-zzwf"
server_key: "10815dffd9b021d32a6238e1cb99074c"
server_key_iv: "ca381059cdc74893a21ee1fa5fbe65da"
server_key_version: "encryptionkey@0"
I20260620 20:22:55.107838 7139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:55.133546 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:55.127978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:35317 (local address 127.0.0.1:41918)
0620 20:22:55.128438 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:55.128462 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:55.128608 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:55.129071 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:55.129077 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:55.129105 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:55.129284 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:55.129293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.130518 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.130524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:55.131633 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:55.131646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.132746 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.132753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:55.132911 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:55.132942 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 20:22:55.133071 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 20:22:55.133206 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":242,"threads_started":1}
I20260620 20:22:55.133586 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:55.128099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41918 (local address 127.31.235.188:35317)
0620 20:22:55.128761 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:55.128765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:55.128780 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:22:55.128837 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:55.128841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:55.128911 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:55.129042 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:55.129418 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.130360 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.131783 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.132559 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.132768 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:55.132957 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:55.133090 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 20:22:55.133212 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":378,"threads_started":1}
I20260620 20:22:55.134553 7172 raft_consensus.cc:1203] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260620 20:22:55.134963 7200 sys_catalog.cc:455] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4efb3f7a9871466ca1c95f235aafdb86. Latest consensus state: current_term: 1 leader_uuid: "4efb3f7a9871466ca1c95f235aafdb86" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: NON_VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: true } } }
I20260620 20:22:55.135214 7200 sys_catalog.cc:458] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [sys.catalog]: This master's current role is: LEARNER
I20260620 20:22:55.135959 7114 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86: attempting to promote NON_VOTER faaa7365b68646198e123e92c3e81b12 to VOTER
I20260620 20:22:55.136535 7114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: false } }
I20260620 20:22:55.138149 7172 raft_consensus.cc:1261] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 1 LEARNER]: Refusing update from remote peer 4efb3f7a9871466ca1c95f235aafdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 20:22:55.138887 6974 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 1 FOLLOWER]: Refusing update from remote peer 4efb3f7a9871466ca1c95f235aafdb86: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 20:22:55.138871 7233 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 20:22:55.139736 7233 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 20:22:55.143069 7233 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, faaa7365b68646198e123e92c3e81b12 (127.31.235.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: false } } }
I20260620 20:22:55.143457 7172 raft_consensus.cc:2932] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, faaa7365b68646198e123e92c3e81b12 (127.31.235.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: false } } }
I20260620 20:22:55.145853 6974 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, faaa7365b68646198e123e92c3e81b12 (127.31.235.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: false } } }
I20260620 20:22:55.151100 7200 sys_catalog.cc:455] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4efb3f7a9871466ca1c95f235aafdb86" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: false } } }
I20260620 20:22:55.151352 7200 sys_catalog.cc:458] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:22:55.151280 7233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4efb3f7a9871466ca1c95f235aafdb86" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: false } } }
I20260620 20:22:55.151600 7233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:55.151911 7139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:55.151785 7236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4efb3f7a9871466ca1c95f235aafdb86" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: false } } }
I20260620 20:22:55.152184 7234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4efb3f7a9871466ca1c95f235aafdb86" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: false } } }
I20260620 20:22:55.153112 7234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:55.153419 7237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4efb3f7a9871466ca1c95f235aafdb86" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: false } } }
I20260620 20:22:55.153752 7237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:22:55.153954 7236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:55.156803 7139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:55.157091 7139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:55.158258 7139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:55.159958 7139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:55.160082 7139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:55.160174 7139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:55.160254 7139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:55.193842 7139 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:37795
I20260620 20:22:55.193930 7346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:37795 every 8 connection(s)
I20260620 20:22:55.195319 7139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:22:55.195730 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 7139
I20260620 20:22:55.196125 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 6561
Jun 20 20:22:55 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.235.130@KRBTEST.COM
I20260620 20:22:55.236513 7351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:55.197195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58142 (local address 127.31.235.130:39559)
0620 20:22:55.208682 (+ 11487us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:55.208691 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:55.208725 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:22:55.208977 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:55.208986 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:55.209313 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:55.211932 (+ 2619us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:55.212257 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.214399 (+ 2142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.216979 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.218075 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.218221 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:55.227108 (+ 8887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:55.227125 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:55.227140 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:55.227190 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:55.231144 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:55.233389 (+ 2245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:55.233398 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:55.233401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:55.233602 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:55.234054 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:55.234059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:55.234063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:55.234442 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:55.234725 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:55.235837 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0620 20:22:55.236089 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11303,"thread_start_us":10950,"threads_started":1}
I20260620 20:22:55.238448 6684 generic_service.cc:196] Checking for leaks (request via RPC)
Jun 20 20:22:55 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986975, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.188@KRBTEST.COM
Jun 20 20:22:55 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986975, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.189@KRBTEST.COM
Jun 20 20:22:55 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986975, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM
I20260620 20:22:55.250994 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:55.200228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:44493 (local address 127.31.235.188:35317)
0620 20:22:55.200460 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:55.200469 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:55.209082 (+ 8613us) server_negotiation.cc:407] Connection header received
0620 20:22:55.225666 (+ 16584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:55.225672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:55.225757 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:55.225858 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:55.231299 (+ 5441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.232381 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.235231 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.235853 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.235947 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:55.243520 (+ 7573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:55.243525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:55.243529 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:55.243561 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:55.246998 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:55.248100 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:55.248107 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:55.248111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:55.248260 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:55.248766 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:55.248771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:55.248784 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:55.249188 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:55.249324 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:55.250412 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0620 20:22:55.250645 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:22:55.252696 7352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:55.199148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:35317 (local address 127.31.235.129:44493)
0620 20:22:55.208899 (+ 9751us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:55.208966 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:55.224539 (+ 15573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:55.227200 (+ 2661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:55.227204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:55.228327 (+ 1123us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:55.230443 (+ 2116us) spinlock_profiling.cc:243] Waited 966 us on lock 0x61200002c628. stack: 00007f5f9ec7966e 00007f5f9ec79430 00007f5f9dd005ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f5f9ec6f0fd 00007f5f9ec6ed1e 00007f5fb1b7c3cd 00007f5fb1b7c36c 00007f5f9fda8757 00007f5fa567d03a 00007f5fa5704dca 00007f5fa5702f68 00007f5fa573682f 00007f5fa573658d
0620 20:22:55.230750 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:55.230767 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.232711 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.232718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:55.235068 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:55.235085 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.236167 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.236173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:55.236595 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:55.237458 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:55.237512 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:55.243222 (+ 5710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:55.247179 (+ 3957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:55.247189 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:55.247206 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:55.247938 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:55.248408 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:55.248414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:55.248419 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:55.248642 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:55.249402 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:55.249409 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:55.249917 (+ 508us) client_negotiation.cc:769] Sending connection context
0620 20:22:55.251473 (+ 1556us) client_negotiation.cc:240] Negotiation successful
0620 20:22:55.251743 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9365,"mutex_wait_us":195,"spinlock_wait_cycles":2223104,"thread_start_us":9817,"threads_started":1}
I20260620 20:22:55.253366 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:55.210653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:39769 (local address 127.31.235.189:33151)
0620 20:22:55.228422 (+ 17769us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:55.228431 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:55.228457 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:22:55.228701 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:55.228709 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:55.228807 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:55.228943 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:55.230973 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.232493 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.238114 (+ 5621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.239022 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.239174 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:55.244970 (+ 5796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:55.244977 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:55.244982 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:55.245024 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:55.248594 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:55.249377 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:55.249382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:55.249385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:55.249551 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:55.249994 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:55.249998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:55.250001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:55.250328 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:55.250442 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:55.252812 (+ 2370us) server_negotiation.cc:299] Negotiation successful
0620 20:22:55.253018 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17585,"thread_start_us":17872,"threads_started":1}
I20260620 20:22:55.253475 7354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:55.210089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:33151 (local address 127.31.235.129:39769)
0620 20:22:55.225021 (+ 14932us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:55.225044 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:55.225163 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:55.229206 (+ 4043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:55.229214 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:55.230001 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:55.230706 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:55.230739 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.232702 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.232709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:55.237376 (+ 4667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:55.237396 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.240487 (+ 3091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.240501 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:55.241243 (+ 742us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:55.241838 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:55.241855 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:55.244732 (+ 2877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:55.248739 (+ 4007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:55.248746 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:55.248749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:55.249245 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:55.249675 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:55.249680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:55.249683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:55.249860 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:55.250517 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:55.250523 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:55.250656 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 20:22:55.250802 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 20:22:55.251317 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14758,"thread_start_us":14335,"threads_started":1}
I20260620 20:22:55.254534 7355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:55.210433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:56485 (local address 127.31.235.190:33833)
0620 20:22:55.221946 (+ 11513us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:55.221953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:55.225373 (+ 3420us) server_negotiation.cc:407] Connection header received
0620 20:22:55.225442 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:55.225447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:55.225546 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:55.225657 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:55.231285 (+ 5628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.232431 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.240056 (+ 7625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.240707 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.240828 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:55.245773 (+ 4945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:55.245782 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:55.245789 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:55.245831 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:55.249580 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:55.250518 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:55.250525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:55.250530 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:55.250686 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:55.251127 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:55.251133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:55.251137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:55.251485 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:55.251622 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:55.253035 (+ 1413us) server_negotiation.cc:299] Negotiation successful
0620 20:22:55.254066 (+ 1031us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11387,"thread_start_us":11244,"threads_started":1}
I20260620 20:22:55.254925 7347 heartbeater.cc:347] Connected to a master server at 127.31.235.188:35317
I20260620 20:22:55.254943 7349 heartbeater.cc:347] Connected to a master server at 127.31.235.189:33151
I20260620 20:22:55.255280 7349 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:55.255287 7347 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:55.252811 7357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:55.216733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.129:56485)
0620 20:22:55.225247 (+ 8514us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:55.225262 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:55.225352 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:55.226360 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:55.226373 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:55.228370 (+ 1997us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:55.231122 (+ 2752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:55.231133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.233241 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.233249 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:55.239903 (+ 6654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:55.239923 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.240959 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.240966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:55.241632 (+ 666us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:55.242410 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:55.242437 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:55.245558 (+ 3121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:55.249739 (+ 4181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:55.249745 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:55.249748 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:55.250370 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:55.250842 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:55.250848 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:55.250851 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:55.250990 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:55.251691 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:55.251695 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:55.251845 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 20:22:55.251983 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 20:22:55.252305 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8393,"mutex_wait_us":6,"thread_start_us":8679,"threads_started":1}
I20260620 20:22:55.256256 7347 heartbeater.cc:511] Master 127.31.235.188:35317 requested a full tablet report, sending...
I20260620 20:22:55.256295 7349 heartbeater.cc:511] Master 127.31.235.189:33151 requested a full tablet report, sending...
I20260620 20:22:55.256327 7348 heartbeater.cc:347] Connected to a master server at 127.31.235.190:33833
I20260620 20:22:55.256433 7348 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:55.256856 7348 heartbeater.cc:511] Master 127.31.235.190:33833 requested a full tablet report, sending...
I20260620 20:22:55.258739 7162 ts_manager.cc:194] Registered new tserver with Master: 838a74503a5f46b39c7553da21104e3d (127.31.235.129:37795)
I20260620 20:22:55.258752 6964 ts_manager.cc:194] Registered new tserver with Master: 838a74503a5f46b39c7553da21104e3d (127.31.235.129:37795)
I20260620 20:22:55.259662 6366 ts_manager.cc:194] Re-registered known tserver with Master: 838a74503a5f46b39c7553da21104e3d (127.31.235.129:37795)
I20260620 20:22:55.260715 6366 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:56485
I20260620 20:22:55.331159 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 6561
I20260620 20:22:55.349193 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:39559
--local_ip_for_outbound_sockets=127.31.235.130
--tserver_master_addrs=127.31.235.190:33833
--webserver_port=44173
--webserver_interface=127.31.235.130
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.130.keytab
--principal=oryx/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.235.190:33833,127.31.235.189:33151
--tserver_master_addrs=127.31.235.190:33833,127.31.235.189:33151,127.31.235.188:35317 with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:22:55.672775 7359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:55.673058 7359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:55.673153 7359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:55.680697 7359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:55.680809 7359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:55.680922 7359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:22:55.691468 7359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.130.keytab
--principal=oryx/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:39559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=44173
--tserver_master_addrs=127.31.235.190:33833,127.31.235.189:33151,127.31.235.188:35317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:55.692865 7359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:55.694286 7359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:55.707607 7365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:55.707561 7364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:55.707641 7367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:55.708786 7359 server_base.cc:1034] running on GCE node
I20260620 20:22:55.709770 7359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:55.710837 7359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:55.712028 7359 hybrid_clock.cc:648] HybridClock initialized: now 1781986975711972 us; error 49 us; skew 500 ppm
I20260620 20:22:55.712370 7359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:55 dist-test-slave-zzwf krb5kdc[6321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986975, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:55.715731 7359 init.cc:377] Logged in from keytab as oryx/127.31.235.130@KRBTEST.COM (short username oryx)
I20260620 20:22:55.717316 7359 webserver.cc:460] Webserver started at http://127.31.235.130:44173/ using document root <none> and password file <none>
I20260620 20:22:55.718091 7359 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:55.718212 7359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:55.723558 7359 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260620 20:22:55.726322 7374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:55.727814 7359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:55.727988 7359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "b23046502b304dbc98a013e3192761b5"
format_stamp: "Formatted at 2026-06-20 20:22:52 on dist-test-slave-zzwf"
server_key: "a098732dd6cf8d54c7cf3aff517b637f"
server_key_iv: "5159278bf4bc664c9696ac52c028dc72"
server_key_version: "encryptionkey@0"
I20260620 20:22:55.728948 7359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:55.746321 7359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:55.750303 7359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:55.750598 7359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:55.751670 7359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:55.753082 7359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:55.753170 7359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:55.753326 7359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:55.753422 7359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:55.788422 7359 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:39559
I20260620 20:22:55.788511 7486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:39559 every 8 connection(s)
I20260620 20:22:55.789988 7359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:22:55.799275 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 7359
I20260620 20:22:55.799636 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 6694
Jun 20 20:22:55 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986975, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.189@KRBTEST.COM
Jun 20 20:22:55 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.235.131@KRBTEST.COM
Jun 20 20:22:55 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986975, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.188@KRBTEST.COM
Jun 20 20:22:55 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986975, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM
I20260620 20:22:55.836452 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:55.792895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:51095 (local address 127.31.235.189:33151)
0620 20:22:55.809013 (+ 16118us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:55.809017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:55.810434 (+ 1417us) server_negotiation.cc:407] Connection header received
0620 20:22:55.814692 (+ 4258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:55.814701 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:55.814825 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:55.814924 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:55.817868 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.819373 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.822457 (+ 3084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.823350 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.823443 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:55.828505 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:55.828514 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:55.828520 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:55.828559 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:55.832249 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:55.833222 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:55.833228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:55.833231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:55.833413 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:55.834256 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:55.834262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:55.834265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:55.834652 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:55.834794 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:55.835818 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0620 20:22:55.836047 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16016,"thread_start_us":16106,"threads_started":1}
I20260620 20:22:55.838878 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:55.793570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:35317 (local address 127.31.235.130:52505)
0620 20:22:55.814224 (+ 20654us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:55.814249 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:55.814386 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:55.816984 (+ 2598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:55.816988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:55.817704 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:55.817977 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:55.817989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.819637 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.819643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:55.822193 (+ 2550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:55.822209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.823518 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.823527 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:55.823842 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:55.824984 (+ 1142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:55.825047 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:55.831176 (+ 6129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:55.835226 (+ 4050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:55.835232 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:55.835238 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:55.835777 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:55.836173 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:55.836179 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:55.836183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:55.836317 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:55.836989 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:55.836994 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:55.837152 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 20:22:55.837313 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 20:22:55.837746 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20447,"mutex_wait_us":531,"spinlock_wait_cycles":27136,"thread_start_us":15730,"threads_started":1}
I20260620 20:22:55.838901 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:55.793570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:33151 (local address 127.31.235.130:51095)
0620 20:22:55.810247 (+ 16677us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:55.810303 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:55.813350 (+ 3047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:55.815509 (+ 2159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:55.815526 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:55.816519 (+ 993us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:55.817619 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:55.817649 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.819642 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.819650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:55.822258 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:55.822276 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.823702 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.823708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:55.823892 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:55.824990 (+ 1098us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:55.825016 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:55.828282 (+ 3266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:55.832426 (+ 4144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:55.832434 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:55.832452 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:55.833024 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:55.833877 (+ 853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:55.833885 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:55.833889 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:55.834122 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:55.834897 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:55.834907 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:55.835373 (+ 466us) client_negotiation.cc:769] Sending connection context
0620 20:22:55.836744 (+ 1371us) client_negotiation.cc:240] Negotiation successful
0620 20:22:55.837361 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16287,"mutex_wait_us":75,"spinlock_wait_cycles":18304,"thread_start_us":20139,"threads_started":1}
I20260620 20:22:55.839756 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:55.811256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.130:42599)
0620 20:22:55.815258 (+ 4002us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:55.815280 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:55.815415 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:55.823948 (+ 8533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:55.823952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:55.824516 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:55.824857 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:55.824871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.826510 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.826516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:55.828014 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:55.828027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.829080 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.829087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:55.829298 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:55.829882 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:55.829911 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:55.832505 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:55.836907 (+ 4402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:55.836914 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:55.836917 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:55.837579 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:55.837983 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:55.837991 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:55.837995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:55.838166 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:55.838755 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:55.838760 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:55.838918 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 20:22:55.839030 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 20:22:55.839343 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3840,"thread_start_us":4221,"threads_started":1}
I20260620 20:22:55.839949 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:55.792892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:52505 (local address 127.31.235.188:35317)
0620 20:22:55.816702 (+ 23810us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:55.816706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:55.816722 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:22:55.816768 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:55.816772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:55.816833 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:55.816964 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:55.818188 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.819424 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.822376 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.823043 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.823168 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:55.831413 (+ 8245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:55.831420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:55.831424 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:55.831473 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:55.835073 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:55.835900 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:55.835906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:55.835909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:55.836037 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:55.836442 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:55.836450 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:55.836453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:55.836801 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:55.836935 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:55.839413 (+ 2478us) server_negotiation.cc:299] Negotiation successful
0620 20:22:55.839627 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23708,"thread_start_us":23595,"threads_started":1}
I20260620 20:22:55.841037 7489 heartbeater.cc:347] Connected to a master server at 127.31.235.189:33151
I20260620 20:22:55.841086 7496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:55.800747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.31.235.131:34255)
0620 20:22:55.809087 (+ 8340us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:55.809104 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:55.817150 (+ 8046us) server_negotiation.cc:407] Connection header received
0620 20:22:55.817243 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:55.817253 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:55.817548 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:55.817702 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:55.819982 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.821606 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.823442 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.824250 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.824378 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:55.832286 (+ 7908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:55.832303 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:55.832323 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:55.832374 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:55.836619 (+ 4245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:55.837527 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:55.837537 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:55.837541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:55.837727 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:55.838171 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:55.838176 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:55.838179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:55.838582 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:55.838856 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:55.840441 (+ 1585us) server_negotiation.cc:299] Negotiation successful
0620 20:22:55.840693 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7704,"thread_start_us":7569,"threads_started":1}
I20260620 20:22:55.841174 7487 heartbeater.cc:347] Connected to a master server at 127.31.235.188:35317
I20260620 20:22:55.841415 7489 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:55.841421 7487 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:55.842252 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:55.811262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:42599 (local address 127.31.235.190:33833)
0620 20:22:55.823520 (+ 12258us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:55.823526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:55.823545 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:22:55.823603 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:55.823622 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:55.823702 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:55.825015 (+ 1313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:55.825179 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.826308 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.828239 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:55.828881 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:55.829632 (+ 751us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:55.832735 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:55.832742 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:55.832746 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:55.832785 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:55.836501 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:55.837686 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:55.837698 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:55.837702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:55.837851 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:55.838307 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:55.838312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:55.838315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:55.838612 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:55.841538 (+ 2926us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:55.841653 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 20:22:55.841904 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12138,"thread_start_us":12521,"threads_started":1}
I20260620 20:22:55.842289 7489 heartbeater.cc:511] Master 127.31.235.189:33151 requested a full tablet report, sending...
I20260620 20:22:55.842435 6817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:55.843037 7488 heartbeater.cc:347] Connected to a master server at 127.31.235.190:33833
I20260620 20:22:55.843192 7488 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:55.843637 7488 heartbeater.cc:511] Master 127.31.235.190:33833 requested a full tablet report, sending...
I20260620 20:22:55.843688 7487 heartbeater.cc:511] Master 127.31.235.188:35317 requested a full tablet report, sending...
I20260620 20:22:55.844808 6964 ts_manager.cc:194] Registered new tserver with Master: b23046502b304dbc98a013e3192761b5 (127.31.235.130:39559)
I20260620 20:22:55.845155 7162 ts_manager.cc:194] Registered new tserver with Master: b23046502b304dbc98a013e3192761b5 (127.31.235.130:39559)
I20260620 20:22:55.845389 6366 ts_manager.cc:194] Re-registered known tserver with Master: b23046502b304dbc98a013e3192761b5 (127.31.235.130:39559)
I20260620 20:22:55.846232 6366 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:42599
I20260620 20:22:55.930648 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 6694
I20260620 20:22:55.947811 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:34255
--local_ip_for_outbound_sockets=127.31.235.131
--tserver_master_addrs=127.31.235.190:33833
--webserver_port=46277
--webserver_interface=127.31.235.131
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.131.keytab
--principal=oryx/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.235.190:33833,127.31.235.189:33151
--tserver_master_addrs=127.31.235.190:33833,127.31.235.189:33151,127.31.235.188:35317 with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:22:56.263388 7348 heartbeater.cc:503] Master 127.31.235.190:33833 was elected leader, sending a full tablet report...
W20260620 20:22:56.295087 7500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:22:56.295408 7500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:22:56.295571 7500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:22:56.303025 7500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:22:56.303177 7500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:22:56.303349 7500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:22:56.313530 7500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.131.keytab
--principal=oryx/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:34255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=46277
--tserver_master_addrs=127.31.235.190:33833,127.31.235.189:33151,127.31.235.188:35317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:22:56.314842 7500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:22:56.316102 7500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:22:56.329105 7508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:56.329205 7506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:22:56.329180 7505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:22:56.330003 7500 server_base.cc:1034] running on GCE node
I20260620 20:22:56.330533 7500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:22:56.331357 7500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:22:56.332578 7500 hybrid_clock.cc:648] HybridClock initialized: now 1781986976332544 us; error 52 us; skew 500 ppm
I20260620 20:22:56.332890 7500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:22:56 dist-test-slave-zzwf krb5kdc[6321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986976, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:22:56.339907 7500 init.cc:377] Logged in from keytab as oryx/127.31.235.131@KRBTEST.COM (short username oryx)
I20260620 20:22:56.341614 7500 webserver.cc:460] Webserver started at http://127.31.235.131:46277/ using document root <none> and password file <none>
I20260620 20:22:56.342285 7500 fs_manager.cc:359] Metadata directory not provided
I20260620 20:22:56.342437 7500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:22:56.347599 7500 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:22:56.350286 7515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:56.351667 7500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:22:56.351871 7500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "e6c9422c263247ba8aae81e971905dc8"
format_stamp: "Formatted at 2026-06-20 20:22:53 on dist-test-slave-zzwf"
server_key: "f93b888329918af2c81b85d7bf7d2260"
server_key_iv: "d3fe348256aa7ad7d8705872507f20e1"
server_key_version: "encryptionkey@0"
I20260620 20:22:56.352890 7500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:22:56.374722 7500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:22:56.378852 7500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:22:56.379174 7500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:22:56.380118 7500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:22:56.381378 7500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:22:56.381487 7500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:56.381644 7500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:22:56.381747 7500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:22:56.413822 7500 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:34255
I20260620 20:22:56.413908 7627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:34255 every 8 connection(s)
I20260620 20:22:56.415201 7500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:22:56.420255 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 7500
Jun 20 20:22:56 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986976, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.188@KRBTEST.COM
Jun 20 20:22:56 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986976, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM
I20260620 20:22:56.470669 7635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.418739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:35317 (local address 127.31.235.131:40629)
0620 20:22:56.438622 (+ 19883us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:56.438682 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:56.442066 (+ 3384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:56.446591 (+ 4525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:56.446609 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:56.447560 (+ 951us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:56.448556 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:56.448588 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.451789 (+ 3201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.451797 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:56.454438 (+ 2641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:56.454453 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.455268 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.455278 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:56.457045 (+ 1767us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:56.457894 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:56.457940 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:56.461368 (+ 3428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:56.465641 (+ 4273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:56.465652 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:56.465674 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:56.466304 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:56.466752 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:56.466759 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:56.466763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:56.466987 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:56.467725 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:56.467734 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:56.468197 (+ 463us) client_negotiation.cc:769] Sending connection context
0620 20:22:56.468517 (+ 320us) client_negotiation.cc:240] Negotiation successful
0620 20:22:56.469077 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18336,"thread_start_us":23840,"threads_started":1}
I20260620 20:22:56.471756 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.418540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:40629 (local address 127.31.235.188:35317)
0620 20:22:56.445706 (+ 27166us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:56.445716 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:56.445742 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:22:56.445814 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:56.445823 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:56.445925 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:56.446054 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:56.448912 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.450369 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.454614 (+ 4245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.455025 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.455165 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:56.461623 (+ 6458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:56.461630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:56.461635 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:56.461673 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:56.465339 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:56.466455 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:56.466461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:56.466465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:56.466616 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:56.467136 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:56.467141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:56.467145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:56.467486 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:56.467635 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:56.471172 (+ 3537us) server_negotiation.cc:299] Negotiation successful
0620 20:22:56.471387 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27004,"thread_start_us":30186,"threads_started":1}
I20260620 20:22:56.473939 7628 heartbeater.cc:347] Connected to a master server at 127.31.235.188:35317
I20260620 20:22:56.474299 7628 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:56.475275 7628 heartbeater.cc:511] Master 127.31.235.188:35317 requested a full tablet report, sending...
I20260620 20:22:56.477121 7162 ts_manager.cc:194] Registered new tserver with Master: e6c9422c263247ba8aae81e971905dc8 (127.31.235.131:34255)
I20260620 20:22:56.478016 7643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.429961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56614 (local address 127.31.235.190:33833)
0620 20:22:56.451692 (+ 21731us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:56.451699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:56.455528 (+ 3829us) server_negotiation.cc:407] Connection header received
0620 20:22:56.455782 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:56.455789 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:56.455869 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:56.455990 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:56.458212 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.459294 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.462294 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.463764 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.463891 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:56.467397 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:56.467402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:56.467407 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:56.467452 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:56.471691 (+ 4239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:56.474592 (+ 2901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:56.474598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:56.474602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:56.474769 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:56.475241 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:56.475246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:56.475250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:56.475603 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:56.475814 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:56.476297 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 20:22:56.477751 (+ 1454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21617,"thread_start_us":23889,"threads_started":1}
I20260620 20:22:56.478360 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.449656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.131:49911)
0620 20:22:56.454013 (+ 4357us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:56.454035 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:56.454143 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:56.454549 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:56.454553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:56.455097 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:56.455483 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:56.455496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.457363 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.457369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:56.459499 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:56.459511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.460429 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.460435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:56.460613 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:56.461249 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:56.461270 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:56.472047 (+ 10777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:56.475694 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:56.475699 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:56.475702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:56.476218 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:56.476650 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:56.476654 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:56.476657 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:56.476800 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:56.477347 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:56.477351 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:56.477513 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 20:22:56.477652 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 20:22:56.477934 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4230,"mutex_wait_us":185,"thread_start_us":4566,"threads_started":1}
I20260620 20:22:56.478863 7640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.443271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:49911 (local address 127.31.235.190:33833)
0620 20:22:56.454088 (+ 10817us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:56.454094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:56.454112 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:22:56.454298 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:56.454303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:56.454385 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:56.454506 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:56.455617 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.456815 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.459627 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.460259 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.460367 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:56.472214 (+ 11847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:56.472222 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:56.472227 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:56.472262 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:56.475591 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:56.476432 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:56.476437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:56.476440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:56.476561 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:56.476931 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:56.476937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:56.476940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:56.477253 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:56.477383 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:56.478304 (+ 921us) server_negotiation.cc:299] Negotiation successful
0620 20:22:56.478493 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10701,"thread_start_us":8224,"threads_started":1}
I20260620 20:22:56.479542 7632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.440232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34580 (local address 127.31.235.189:33151)
0620 20:22:56.450953 (+ 10721us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:56.450962 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:56.454154 (+ 3192us) server_negotiation.cc:407] Connection header received
0620 20:22:56.454224 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:56.454228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:56.454297 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:56.454391 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:56.456893 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.459946 (+ 3053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.462179 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.462938 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.463030 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:56.468023 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:56.468032 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:56.468040 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:56.468090 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:56.472045 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:56.473453 (+ 1408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:56.473462 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:56.473468 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:56.473669 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:56.476520 (+ 2851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:56.476529 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:56.476534 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:56.476911 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:56.477153 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:56.478447 (+ 1294us) server_negotiation.cc:299] Negotiation successful
0620 20:22:56.478728 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10560,"thread_start_us":7328,"threads_started":1}
I20260620 20:22:56.479591 7629 heartbeater.cc:347] Connected to a master server at 127.31.235.190:33833
I20260620 20:22:56.479709 7629 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:56.480216 7629 heartbeater.cc:511] Master 127.31.235.190:33833 requested a full tablet report, sending...
I20260620 20:22:56.481329 6366 ts_manager.cc:194] Re-registered known tserver with Master: e6c9422c263247ba8aae81e971905dc8 (127.31.235.131:34255)
I20260620 20:22:56.482450 6366 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:49911
Jun 20 20:22:56 dist-test-slave-zzwf krb5kdc[6321](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986976, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.189@KRBTEST.COM
I20260620 20:22:56.491498 7642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.418193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:36439 (local address 127.31.235.189:33151)
0620 20:22:56.447870 (+ 29677us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:56.447879 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:56.447905 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:22:56.447978 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:56.447986 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:56.448081 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:56.450998 (+ 2917us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:56.451210 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.452551 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.459646 (+ 7095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.460297 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.478831 (+ 18534us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:56.484319 (+ 5488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:56.484325 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:56.484329 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:56.484366 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:56.488489 (+ 4123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:56.489425 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:56.489430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:56.489434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:56.489604 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:56.490005 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:56.490010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:56.490013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:56.490349 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:56.490494 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:56.490899 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 20:22:56.491094 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29514,"thread_start_us":37512,"threads_started":1}
I20260620 20:22:56.491601 7634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.418739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:33151 (local address 127.31.235.131:36439)
0620 20:22:56.442974 (+ 24235us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:56.442993 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:56.443101 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:56.449704 (+ 6603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:56.449709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:56.450412 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:22:56.450772 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:56.450788 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.456929 (+ 6141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.456934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:56.459118 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:56.459131 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.479101 (+ 19970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.479108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:56.479300 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:56.479839 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:22:56.479861 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:22:56.484090 (+ 4229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:22:56.488654 (+ 4564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:56.488660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:56.488665 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:56.489250 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:56.489752 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:22:56.489757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:22:56.489760 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:22:56.489911 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:22:56.490504 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:22:56.490511 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:22:56.490645 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 20:22:56.490794 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 20:22:56.491095 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24081,"mutex_wait_us":140,"spinlock_wait_cycles":55040,"thread_start_us":18613,"threads_started":1}
I20260620 20:22:56.492794 7630 heartbeater.cc:347] Connected to a master server at 127.31.235.189:33151
I20260620 20:22:56.492944 7630 heartbeater.cc:464] Registering TS with master...
I20260620 20:22:56.493310 7630 heartbeater.cc:511] Master 127.31.235.189:33151 requested a full tablet report, sending...
I20260620 20:22:56.494356 6964 ts_manager.cc:194] Registered new tserver with Master: e6c9422c263247ba8aae81e971905dc8 (127.31.235.131:34255)
I20260620 20:22:56.496825 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.463343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37580 (local address 127.31.235.188:35317)
0620 20:22:56.472849 (+ 9506us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:56.472856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:56.480052 (+ 7196us) server_negotiation.cc:407] Connection header received
0620 20:22:56.480106 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:56.480113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:56.480203 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:56.480294 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:56.482188 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.483249 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.485033 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.485730 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.485866 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:56.490030 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:56.490037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:56.490042 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:56.490083 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:56.493903 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:56.494788 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:56.494794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:56.494798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:56.494948 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:56.495328 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:56.495334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:56.495338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:56.495722 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:56.495966 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:56.496194 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 20:22:56.496432 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8661,"thread_start_us":19142,"threads_started":1}
I20260620 20:22:56.487685 6366 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:22:56.498519 6366 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:22:56.527504 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.515053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:37795 (local address 127.0.0.1:44238)
0620 20:22:56.517992 (+ 2939us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:56.518018 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:56.518181 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:56.519346 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:56.519351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:56.519375 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:56.519797 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:56.519811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.521971 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.521978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:56.524146 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:56.524165 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.526736 (+ 2571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.526744 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:56.526914 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:56.526968 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 20:22:56.527041 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 20:22:56.527171 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2754,"thread_start_us":2604,"threads_started":1}
I20260620 20:22:56.528697 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.515614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:34255 (local address 127.0.0.1:49236)
0620 20:22:56.520315 (+ 4701us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:56.520333 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:56.520454 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:56.521197 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:56.521203 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:56.521227 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:56.521545 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:56.521556 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.523949 (+ 2393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.523960 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:56.526066 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:56.526080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.527958 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.527964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:56.528151 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:56.528181 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 20:22:56.528238 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 20:22:56.528358 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4582,"thread_start_us":4914,"threads_started":1}
I20260620 20:22:56.529742 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.515780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49236 (local address 127.31.235.131:34255)
0620 20:22:56.519564 (+ 3784us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:56.519578 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:56.520572 (+ 994us) server_negotiation.cc:407] Connection header received
0620 20:22:56.520772 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:56.520782 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:56.520995 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:56.521788 (+ 793us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:56.522167 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.523803 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.526283 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.527739 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.528935 (+ 1196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:56.529108 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:56.529209 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 20:22:56.529376 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3555,"thread_start_us":3347,"threads_started":1}
I20260620 20:22:56.529949 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.515979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44238 (local address 127.31.235.129:37795)
0620 20:22:56.516590 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:56.516604 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:56.518335 (+ 1731us) server_negotiation.cc:407] Connection header received
0620 20:22:56.518473 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:56.518486 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:56.518861 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:56.519096 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:56.519962 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.521817 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.524676 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.526510 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.528440 (+ 1930us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:56.529189 (+ 749us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:56.529248 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 20:22:56.529404 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":265,"threads_started":1}
I20260620 20:22:56.533322 7282 tablet_service.cc:1467] Processing CreateTablet for tablet dc7267b24c264a1cb72018c86406dbf2 (DEFAULT_TABLE table=test-table [id=024c720d8f694f5ebe8f70c160955dfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:56.534647 7282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7267b24c264a1cb72018c86406dbf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:56.545358 7563 tablet_service.cc:1467] Processing CreateTablet for tablet dc7267b24c264a1cb72018c86406dbf2 (DEFAULT_TABLE table=test-table [id=024c720d8f694f5ebe8f70c160955dfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:56.546833 7563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7267b24c264a1cb72018c86406dbf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:56.547405 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.521699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:39559 (local address 127.0.0.1:50196)
0620 20:22:56.527743 (+ 6044us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:56.527762 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:56.527860 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:56.541040 (+ 13180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:56.541045 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:56.541074 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:56.541264 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:56.541275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.543084 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.543091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:56.544653 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:56.544667 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.546548 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.546555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:56.546751 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:56.546795 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 20:22:56.546928 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 20:22:56.547084 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5928,"thread_start_us":19749,"threads_started":1}
I20260620 20:22:56.547480 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.521106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50196 (local address 127.31.235.130:39559)
0620 20:22:56.540247 (+ 19141us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:56.540260 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:56.540303 (+ 43us) server_negotiation.cc:407] Connection header received
0620 20:22:56.540434 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:56.540447 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:56.540778 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:56.541058 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:56.541451 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.542914 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.544831 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.546369 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.546569 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:56.546752 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:56.546971 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 20:22:56.547205 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18946,"thread_start_us":18844,"threads_started":1}
I20260620 20:22:56.550223 7664 tablet_bootstrap.cc:492] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d: Bootstrap starting.
I20260620 20:22:56.551084 7422 tablet_service.cc:1467] Processing CreateTablet for tablet dc7267b24c264a1cb72018c86406dbf2 (DEFAULT_TABLE table=test-table [id=024c720d8f694f5ebe8f70c160955dfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:56.552842 7422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7267b24c264a1cb72018c86406dbf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:56.554068 7664 tablet_bootstrap.cc:654] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:56.555397 7664 log.cc:824] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:56.558671 7664 tablet_bootstrap.cc:492] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d: No bootstrap required, opened a new log
I20260620 20:22:56.558954 7664 ts_tablet_manager.cc:1397] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 20:22:56.561889 7666 tablet_bootstrap.cc:492] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8: Bootstrap starting.
I20260620 20:22:56.564749 7666 tablet_bootstrap.cc:654] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:56.565747 7666 log.cc:824] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:56.569253 7668 tablet_bootstrap.cc:492] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5: Bootstrap starting.
I20260620 20:22:56.570340 7664 raft_consensus.cc:348] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838a74503a5f46b39c7553da21104e3d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37795 } } peers { permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 } } peers { permanent_uuid: "b23046502b304dbc98a013e3192761b5" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39559 } }
I20260620 20:22:56.570724 7664 raft_consensus.cc:374] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:56.570833 7664 raft_consensus.cc:729] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 838a74503a5f46b39c7553da21104e3d, State: Initialized, Role: FOLLOWER
I20260620 20:22:56.571506 7664 consensus_queue.cc:260] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838a74503a5f46b39c7553da21104e3d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37795 } } peers { permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 } } peers { permanent_uuid: "b23046502b304dbc98a013e3192761b5" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39559 } }
I20260620 20:22:56.572793 7668 tablet_bootstrap.cc:654] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:56.573941 7668 log.cc:824] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5: Log is configured to *not* fsync() on all Append() calls
I20260620 20:22:56.576236 7664 ts_tablet_manager.cc:1428] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d: Time spent starting tablet: real 0.017s user 0.010s sys 0.006s
I20260620 20:22:56.581139 7666 tablet_bootstrap.cc:492] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8: No bootstrap required, opened a new log
I20260620 20:22:56.581418 7666 ts_tablet_manager.cc:1397] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.005s
I20260620 20:22:56.587913 7668 tablet_bootstrap.cc:492] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5: No bootstrap required, opened a new log
I20260620 20:22:56.588181 7668 ts_tablet_manager.cc:1397] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.012s
I20260620 20:22:56.591800 7666 raft_consensus.cc:348] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838a74503a5f46b39c7553da21104e3d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37795 } } peers { permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 } } peers { permanent_uuid: "b23046502b304dbc98a013e3192761b5" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39559 } }
I20260620 20:22:56.592213 7666 raft_consensus.cc:374] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:56.592314 7666 raft_consensus.cc:729] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c9422c263247ba8aae81e971905dc8, State: Initialized, Role: FOLLOWER
I20260620 20:22:56.592922 7666 consensus_queue.cc:260] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838a74503a5f46b39c7553da21104e3d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37795 } } peers { permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 } } peers { permanent_uuid: "b23046502b304dbc98a013e3192761b5" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39559 } }
I20260620 20:22:56.594668 7666 ts_tablet_manager.cc:1428] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260620 20:22:56.594985 7629 heartbeater.cc:503] Master 127.31.235.190:33833 was elected leader, sending a full tablet report...
I20260620 20:22:56.595590 7668 raft_consensus.cc:348] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838a74503a5f46b39c7553da21104e3d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37795 } } peers { permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 } } peers { permanent_uuid: "b23046502b304dbc98a013e3192761b5" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39559 } }
I20260620 20:22:56.596058 7668 raft_consensus.cc:374] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:56.596196 7668 raft_consensus.cc:729] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23046502b304dbc98a013e3192761b5, State: Initialized, Role: FOLLOWER
I20260620 20:22:56.596768 7668 consensus_queue.cc:260] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838a74503a5f46b39c7553da21104e3d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37795 } } peers { permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 } } peers { permanent_uuid: "b23046502b304dbc98a013e3192761b5" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39559 } }
I20260620 20:22:56.599162 7668 ts_tablet_manager.cc:1428] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260620 20:22:56.599495 7488 heartbeater.cc:503] Master 127.31.235.190:33833 was elected leader, sending a full tablet report...
W20260620 20:22:56.673610 7631 tablet.cc:2369] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:22:56.741691 7353 tablet.cc:2369] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:22:56.798866 7490 tablet.cc:2369] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:22:56.851554 7669 raft_consensus.cc:482] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:22:56.851872 7669 raft_consensus.cc:504] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838a74503a5f46b39c7553da21104e3d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37795 } } peers { permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 } } peers { permanent_uuid: "b23046502b304dbc98a013e3192761b5" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39559 } }
I20260620 20:22:56.853642 7669 leader_election.cc:288] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6c9422c263247ba8aae81e971905dc8 (127.31.235.131:34255), b23046502b304dbc98a013e3192761b5 (127.31.235.130:39559)
I20260620 20:22:56.860059 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.854053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:48039 (local address 127.31.235.131:34255)
0620 20:22:56.854283 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:56.854291 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:56.854872 (+ 581us) server_negotiation.cc:407] Connection header received
0620 20:22:56.854956 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:56.854961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:56.855030 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:56.855128 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:56.855611 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.856335 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.858243 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.859121 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.859303 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:56.859373 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:56.859685 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 20:22:56.859803 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:22:56.860131 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.853858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:34255 (local address 127.31.235.129:48039)
0620 20:22:56.854463 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:56.854479 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:56.854655 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:56.855156 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:56.855160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:56.855195 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:56.855478 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:56.855486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.856460 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.856467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:56.858080 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:56.858092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.859317 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.859323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:56.859535 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:56.859559 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:22:56.859663 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 20:22:56.859798 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":304,"threads_started":1}
I20260620 20:22:56.860500 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.854764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:36503 (local address 127.31.235.130:39559)
0620 20:22:56.854929 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:56.854935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:56.855212 (+ 277us) server_negotiation.cc:407] Connection header received
0620 20:22:56.855405 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:56.855413 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:56.855472 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:56.855582 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:56.856045 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.856860 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.858712 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.859624 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.859787 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:56.859860 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:56.860139 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 20:22:56.860257 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 20:22:56.860559 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.854550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:39559 (local address 127.31.235.129:36503)
0620 20:22:56.855081 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:56.855094 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:56.855254 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:56.855595 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:56.855599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:56.855619 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:56.855906 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:56.855913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.856988 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.856994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:56.858572 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:56.858583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.859825 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.859831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:56.859983 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:56.860005 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:22:56.860120 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 20:22:56.860240 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":290,"threads_started":1}
I20260620 20:22:56.861253 7583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7267b24c264a1cb72018c86406dbf2" candidate_uuid: "838a74503a5f46b39c7553da21104e3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c9422c263247ba8aae81e971905dc8" is_pre_election: true
I20260620 20:22:56.861635 7442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7267b24c264a1cb72018c86406dbf2" candidate_uuid: "838a74503a5f46b39c7553da21104e3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23046502b304dbc98a013e3192761b5" is_pre_election: true
I20260620 20:22:56.861754 7583 raft_consensus.cc:2454] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 838a74503a5f46b39c7553da21104e3d in term 0.
I20260620 20:22:56.862087 7442 raft_consensus.cc:2454] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 838a74503a5f46b39c7553da21104e3d in term 0.
I20260620 20:22:56.862329 7229 leader_election.cc:302] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 838a74503a5f46b39c7553da21104e3d, e6c9422c263247ba8aae81e971905dc8; no voters:
I20260620 20:22:56.862694 7669 raft_consensus.cc:2781] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:22:56.862900 7669 raft_consensus.cc:482] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:22:56.863024 7669 raft_consensus.cc:3037] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:56.865331 7669 raft_consensus.cc:504] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838a74503a5f46b39c7553da21104e3d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37795 } } peers { permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 } } peers { permanent_uuid: "b23046502b304dbc98a013e3192761b5" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39559 } }
I20260620 20:22:56.865895 7669 leader_election.cc:288] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d [CANDIDATE]: Term 1 election: Requested vote from peers e6c9422c263247ba8aae81e971905dc8 (127.31.235.131:34255), b23046502b304dbc98a013e3192761b5 (127.31.235.130:39559)
I20260620 20:22:56.866444 7442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7267b24c264a1cb72018c86406dbf2" candidate_uuid: "838a74503a5f46b39c7553da21104e3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23046502b304dbc98a013e3192761b5"
I20260620 20:22:56.866463 7583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7267b24c264a1cb72018c86406dbf2" candidate_uuid: "838a74503a5f46b39c7553da21104e3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c9422c263247ba8aae81e971905dc8"
I20260620 20:22:56.866649 7583 raft_consensus.cc:3037] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:56.866636 7442 raft_consensus.cc:3037] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:56.868769 7442 raft_consensus.cc:2454] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 838a74503a5f46b39c7553da21104e3d in term 1.
I20260620 20:22:56.868791 7583 raft_consensus.cc:2454] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 838a74503a5f46b39c7553da21104e3d in term 1.
I20260620 20:22:56.869150 7229 leader_election.cc:302] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 838a74503a5f46b39c7553da21104e3d, b23046502b304dbc98a013e3192761b5; no voters:
I20260620 20:22:56.869457 7669 raft_consensus.cc:2781] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:56.869940 7669 raft_consensus.cc:686] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d [term 1 LEADER]: Becoming Leader. State: Replica: 838a74503a5f46b39c7553da21104e3d, State: Running, Role: LEADER
I20260620 20:22:56.870483 7669 consensus_queue.cc:237] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838a74503a5f46b39c7553da21104e3d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37795 } } peers { permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 } } peers { permanent_uuid: "b23046502b304dbc98a013e3192761b5" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39559 } }
I20260620 20:22:56.875098 6365 catalog_manager.cc:5495] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d reported cstate change: term changed from 0 to 1, leader changed from <none> to 838a74503a5f46b39c7553da21104e3d (127.31.235.129). New cstate: current_term: 1 leader_uuid: "838a74503a5f46b39c7553da21104e3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "838a74503a5f46b39c7553da21104e3d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b23046502b304dbc98a013e3192761b5" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39559 } health_report { overall_health: UNKNOWN } } }
I20260620 20:22:56.897076 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.892361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44244 (local address 127.31.235.129:37795)
0620 20:22:56.892534 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:56.892538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:56.892635 (+ 97us) server_negotiation.cc:407] Connection header received
0620 20:22:56.892790 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:56.892794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:56.892874 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:56.892996 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:56.893303 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.894062 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.895336 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.895717 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.895803 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:56.896178 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:56.896646 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:56.896758 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:56.896829 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 20:22:56.896886 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:22:56.904982 7442 raft_consensus.cc:1261] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5 [term 1 FOLLOWER]: Refusing update from remote peer 838a74503a5f46b39c7553da21104e3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:56.904981 7583 raft_consensus.cc:1261] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8 [term 1 FOLLOWER]: Refusing update from remote peer 838a74503a5f46b39c7553da21104e3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:56.905962 7675 consensus_queue.cc:1035] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:56.906340 7669 consensus_queue.cc:1035] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b23046502b304dbc98a013e3192761b5" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:56.917825 7679 mvcc.cc:205] Tried to move back new op lower bound from 7299018657392504832 to 7299018657264930816. Current Snapshot: MvccSnapshot[applied={T|T < 7299018657392504832}]
I20260620 20:22:56.933773 7680 mvcc.cc:205] Tried to move back new op lower bound from 7299018657392504832 to 7299018657264930816. Current Snapshot: MvccSnapshot[applied={T|T < 7299018657392504832}]
I20260620 20:22:56.950907 6365 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:56.951181 6365 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:56.957365 6365 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4efb3f7a9871466ca1c95f235aafdb86: Sending DeleteTablet for 3 replicas of tablet dc7267b24c264a1cb72018c86406dbf2
I20260620 20:22:56.958506 7282 tablet_service.cc:1514] Processing DeleteTablet for tablet dc7267b24c264a1cb72018c86406dbf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:56 UTC) from {username='oryx', principal='oryx/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:44238
I20260620 20:22:56.958721 7422 tablet_service.cc:1514] Processing DeleteTablet for tablet dc7267b24c264a1cb72018c86406dbf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:56 UTC) from {username='oryx', principal='oryx/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:50196
I20260620 20:22:56.959617 7563 tablet_service.cc:1514] Processing DeleteTablet for tablet dc7267b24c264a1cb72018c86406dbf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:56 UTC) from {username='oryx', principal='oryx/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:49236
I20260620 20:22:56.974232 7690 tablet_replica.cc:330] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5: stopping tablet replica
I20260620 20:22:56.974845 7690 raft_consensus.cc:2229] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:22:56.975440 7690 raft_consensus.cc:2258] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:56.977669 7690 ts_tablet_manager.cc:1905] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:56.981959 7691 tablet_replica.cc:330] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8: stopping tablet replica
I20260620 20:22:56.983065 7691 raft_consensus.cc:2229] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:22:56.984867 7691 raft_consensus.cc:2258] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:56.984992 7690 ts_tablet_manager.cc:1918] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:22:56.985162 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 6334
I20260620 20:22:56.985219 7690 log.cc:1196] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/dc7267b24c264a1cb72018c86406dbf2
I20260620 20:22:56.985895 7690 ts_tablet_manager.cc:1939] T dc7267b24c264a1cb72018c86406dbf2 P b23046502b304dbc98a013e3192761b5: Deleting consensus metadata
I20260620 20:22:56.986275 6396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:56.986574 7689 tablet_replica.cc:330] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d: stopping tablet replica
I20260620 20:22:56.986997 7689 raft_consensus.cc:2229] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:22:56.987716 7689 raft_consensus.cc:2258] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:56.990140 7689 ts_tablet_manager.cc:1905] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:56.990849 7691 ts_tablet_manager.cc:1905] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:56.993984 6352 catalog_manager.cc:4841] TS b23046502b304dbc98a013e3192761b5 (127.31.235.130:39559): tablet dc7267b24c264a1cb72018c86406dbf2 (table test-table [id=024c720d8f694f5ebe8f70c160955dfd]) successfully deleted
I20260620 20:22:57.003516 7642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.961948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34582 (local address 127.31.235.189:33151)
0620 20:22:56.962151 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:56.962160 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:56.983808 (+ 21648us) server_negotiation.cc:407] Connection header received
0620 20:22:56.985225 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:56.985231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:56.985333 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:56.985417 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:56.989557 (+ 4140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.990664 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.992570 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:56.993213 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:56.993302 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:56.996818 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:56.996824 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:56.996828 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:56.996865 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:57.000495 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:57.001270 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:57.001275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:57.001279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:57.001409 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:57.001841 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:57.001846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:57.001849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:57.002191 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:57.002400 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:57.002952 (+ 552us) server_negotiation.cc:299] Negotiation successful
0620 20:22:57.003164 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:22:57.005941 7691 ts_tablet_manager.cc:1918] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:22:57.006139 7691 log.cc:1196] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/dc7267b24c264a1cb72018c86406dbf2
I20260620 20:22:57.006696 7691 ts_tablet_manager.cc:1939] T dc7267b24c264a1cb72018c86406dbf2 P e6c9422c263247ba8aae81e971905dc8: Deleting consensus metadata
I20260620 20:22:57.008396 7689 ts_tablet_manager.cc:1918] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:22:57.008617 7689 log.cc:1196] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/dc7267b24c264a1cb72018c86406dbf2
I20260620 20:22:57.009255 7689 ts_tablet_manager.cc:1939] T dc7267b24c264a1cb72018c86406dbf2 P 838a74503a5f46b39c7553da21104e3d: Deleting consensus metadata
I20260620 20:22:57.019948 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:56.984071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37594 (local address 127.31.235.188:35317)
0620 20:22:56.984292 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:56.984298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:57.004887 (+ 20589us) server_negotiation.cc:407] Connection header received
0620 20:22:57.005131 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:57.005136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:57.005223 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:57.005293 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:57.007321 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:57.008386 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:57.011221 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:57.011846 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:57.011985 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:57.014757 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:57.014771 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:57.014774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:57.014804 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:57.017357 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:57.018004 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:57.018010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:57.018014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:57.018133 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:57.018500 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:57.018505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:57.018508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:57.018792 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:57.019037 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:57.019337 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 20:22:57.019541 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 20:22:57.166613 6350 catalog_manager.cc:4841] TS 838a74503a5f46b39c7553da21104e3d (127.31.235.129:37795): tablet dc7267b24c264a1cb72018c86406dbf2 (table test-table [id=024c720d8f694f5ebe8f70c160955dfd]) successfully deleted
I20260620 20:22:57.167248 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 6334
I20260620 20:22:57.226249 7642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:57.198413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34584 (local address 127.31.235.189:33151)
0620 20:22:57.198578 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:57.198584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:57.209428 (+ 10844us) server_negotiation.cc:407] Connection header received
0620 20:22:57.209768 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:57.209774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:57.209861 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:57.209938 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:57.212462 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:57.213663 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:57.215611 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:57.216296 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:57.216405 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:57.219354 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:57.219361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:57.219365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:57.219402 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:57.223070 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:57.223868 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:57.223874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:57.223878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:57.224042 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:57.224534 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:57.224541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:57.224545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:57.224911 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:57.225181 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:57.225618 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 20:22:57.225881 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:22:57.228415 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:57.209973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37600 (local address 127.31.235.188:35317)
0620 20:22:57.210141 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:57.210148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:57.210688 (+ 540us) server_negotiation.cc:407] Connection header received
0620 20:22:57.210961 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:57.210969 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:57.211080 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:57.211164 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:57.213433 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:57.214593 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:57.216496 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:57.217166 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:57.217295 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:57.222254 (+ 4959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:57.222262 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:57.222266 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:57.222307 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:57.225706 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:57.226447 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:57.226454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:57.226459 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:57.226602 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:57.226934 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:57.226942 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:57.226948 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:57.227316 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:57.227581 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:57.227906 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 20:22:57.228112 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:22:57.934593 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:57.933857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.129:43307)
0620 20:22:57.934320 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:57.934428 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":236,"threads_started":1}
W20260620 20:22:57.934983 7348 heartbeater.cc:644] Failed to heartbeat to 127.31.235.190:33833 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
I20260620 20:22:57.942301 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:57.941341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.130:53647)
0620 20:22:57.941914 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:57.942069 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":315,"threads_started":1}
W20260620 20:22:57.942751 7488 heartbeater.cc:644] Failed to heartbeat to 127.31.235.190:33833 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
I20260620 20:22:57.945108 7702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:57.944308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.131:42275)
0620 20:22:57.944798 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:57.944922 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":268,"threads_started":1}
W20260620 20:22:57.945572 7629 heartbeater.cc:644] Failed to heartbeat to 127.31.235.190:33833 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
I20260620 20:22:58.457252 7703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4efb3f7a9871466ca1c95f235aafdb86)
I20260620 20:22:58.457551 7703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: false } }
I20260620 20:22:58.457783 7704 raft_consensus.cc:482] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4efb3f7a9871466ca1c95f235aafdb86)
I20260620 20:22:58.458048 7704 raft_consensus.cc:504] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: false } }
I20260620 20:22:58.458614 7703 leader_election.cc:288] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4efb3f7a9871466ca1c95f235aafdb86 (127.31.235.190:33833), faaa7365b68646198e123e92c3e81b12 (127.31.235.188:35317)
I20260620 20:22:58.460870 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.459035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.0.0.1:57060)
0620 20:22:58.460372 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.460571 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1241,"thread_start_us":1101,"threads_started":1}
W20260620 20:22:58.461272 7148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
I20260620 20:22:58.461611 7704 leader_election.cc:288] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4efb3f7a9871466ca1c95f235aafdb86 (127.31.235.190:33833), 3799df284b594f518505074efea61789 (127.31.235.189:33151)
I20260620 20:22:58.464303 7707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.463458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.0.0.1:57064)
0620 20:22:58.463781 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.463958 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260620 20:22:58.464542 7705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.458787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.0.0.1:57050)
0620 20:22:58.464062 (+ 5275us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.464279 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5172,"thread_start_us":5069,"threads_started":1}
W20260620 20:22:58.464627 7148 leader_election.cc:334] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4efb3f7a9871466ca1c95f235aafdb86 (127.31.235.190:33833): Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
W20260620 20:22:58.465246 6950 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
I20260620 20:22:58.468067 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.466956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.0.0.1:57080)
0620 20:22:58.467685 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.467853 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":483,"threads_started":1}
W20260620 20:22:58.468312 6950 leader_election.cc:334] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4efb3f7a9871466ca1c95f235aafdb86 (127.31.235.190:33833): Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
I20260620 20:22:58.469575 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.460602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:33151 (local address 127.0.0.1:34586)
0620 20:22:58.461225 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.461246 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:58.461371 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:58.462678 (+ 1307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:58.462682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:58.462712 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:58.463035 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:58.463050 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.464834 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.464839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:58.467041 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:58.467059 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.468829 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.468835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:58.469011 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:58.469043 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 20:22:58.469159 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 20:22:58.469296 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":2948,"threads_started":1}
I20260620 20:22:58.469609 7708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.461412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34586 (local address 127.31.235.189:33151)
0620 20:22:58.462298 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:58.462306 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:58.462330 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:22:58.462405 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:58.462412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:58.462511 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:58.463256 (+ 745us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:58.463452 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.464655 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.467221 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.468686 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.468805 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:58.468853 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:58.469182 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 20:22:58.469290 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":728,"thread_start_us":474,"threads_started":1}
I20260620 20:22:58.470199 6973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "faaa7365b68646198e123e92c3e81b12" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3799df284b594f518505074efea61789" is_pre_election: true
I20260620 20:22:58.470546 6973 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faaa7365b68646198e123e92c3e81b12 in term 1.
I20260620 20:22:58.471020 7149 leader_election.cc:302] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3799df284b594f518505074efea61789, faaa7365b68646198e123e92c3e81b12; no voters: 4efb3f7a9871466ca1c95f235aafdb86
I20260620 20:22:58.471334 7704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 20:22:58.471454 7704 raft_consensus.cc:482] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4efb3f7a9871466ca1c95f235aafdb86)
I20260620 20:22:58.471583 7704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:22:58.472488 7705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.464978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:35317 (local address 127.0.0.1:37612)
0620 20:22:58.465203 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.465236 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:58.465435 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:58.466409 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:58.466414 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:58.466441 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:58.466673 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:58.466683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.468259 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.468265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:58.470366 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:58.470379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.471770 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.471776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:58.471963 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:58.471985 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:22:58.472090 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 20:22:58.472220 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":24}
I20260620 20:22:58.472640 7710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.464434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37612 (local address 127.31.235.188:35317)
0620 20:22:58.466095 (+ 1661us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:58.466102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:58.466125 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:22:58.466180 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:58.466185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:58.466260 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:58.466370 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:58.466823 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.468078 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.470525 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.471595 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.471779 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:58.471851 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:58.472116 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 20:22:58.472240 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1514,"thread_start_us":1371,"threads_started":1}
I20260620 20:22:58.474057 7172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3799df284b594f518505074efea61789" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "faaa7365b68646198e123e92c3e81b12" is_pre_election: true
I20260620 20:22:58.474179 7704 raft_consensus.cc:504] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: false } }
I20260620 20:22:58.474570 7172 raft_consensus.cc:2379] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3799df284b594f518505074efea61789 in current term 2: Already voted for candidate faaa7365b68646198e123e92c3e81b12 in this term.
I20260620 20:22:58.475050 6950 leader_election.cc:302] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3799df284b594f518505074efea61789; no voters: 4efb3f7a9871466ca1c95f235aafdb86, faaa7365b68646198e123e92c3e81b12
I20260620 20:22:58.475119 7704 leader_election.cc:288] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [CANDIDATE]: Term 2 election: Requested vote from peers 4efb3f7a9871466ca1c95f235aafdb86 (127.31.235.190:33833), 3799df284b594f518505074efea61789 (127.31.235.189:33151)
I20260620 20:22:58.475298 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.474840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.0.0.1:57086)
0620 20:22:58.474994 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.475135 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 20:22:58.475425 7703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:22:58.475581 6973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "faaa7365b68646198e123e92c3e81b12" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3799df284b594f518505074efea61789"
I20260620 20:22:58.477438 7703 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:22:58.478410 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.478089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.0.0.1:57088)
0620 20:22:58.478233 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.478317 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 20:22:58.478569 7148 leader_election.cc:334] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4efb3f7a9871466ca1c95f235aafdb86 (127.31.235.190:33833): Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
I20260620 20:22:58.479707 6973 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faaa7365b68646198e123e92c3e81b12 in term 2.
I20260620 20:22:58.480391 7149 leader_election.cc:302] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3799df284b594f518505074efea61789, faaa7365b68646198e123e92c3e81b12; no voters: 4efb3f7a9871466ca1c95f235aafdb86
I20260620 20:22:58.480666 7704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:22:58.481133 7704 raft_consensus.cc:686] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [term 2 LEADER]: Becoming Leader. State: Replica: faaa7365b68646198e123e92c3e81b12, State: Running, Role: LEADER
I20260620 20:22:58.481560 7704 consensus_queue.cc:237] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: false } }
I20260620 20:22:58.483179 7713 sys_catalog.cc:455] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader faaa7365b68646198e123e92c3e81b12. Latest consensus state: current_term: 2 leader_uuid: "faaa7365b68646198e123e92c3e81b12" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: false } } }
I20260620 20:22:58.483743 7713 sys_catalog.cc:458] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:58.484391 7715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:22:58.488024 7715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:22:58.488850 7715 catalog_manager.cc:1259] Loaded cluster ID: 66a41b8acf634ce197e6bc9780a4c844
I20260620 20:22:58.488916 7715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:22:58.490386 7715 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:22:58.491169 7715 catalog_manager.cc:5879] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: Loaded TSK: 0
I20260620 20:22:58.491958 7715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:22:58.515144 7162 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:22:58.522167 7162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:22:58.526536 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.526031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.0.0.1:57100)
0620 20:22:58.526264 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.526360 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 20:22:58.526510 6973 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [term 2 FOLLOWER]: Refusing update from remote peer faaa7365b68646198e123e92c3e81b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 20:22:58.527366 7713 consensus_queue.cc:1035] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 20:22:58.527500 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.527157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.0.0.1:57102)
0620 20:22:58.527314 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.527384 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 20:22:58.527993 7148 consensus_peers.cc:487] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 -> Peer 4efb3f7a9871466ca1c95f235aafdb86 (127.31.235.190:33833): Couldn't send request to peer 4efb3f7a9871466ca1c95f235aafdb86. Status: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 20:22:58.530473 7703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader faaa7365b68646198e123e92c3e81b12. Latest consensus state: current_term: 2 leader_uuid: "faaa7365b68646198e123e92c3e81b12" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: false } } }
I20260620 20:22:58.530812 7703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:22:58.532721 7713 sys_catalog.cc:455] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "faaa7365b68646198e123e92c3e81b12" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: false } } }
I20260620 20:22:58.532819 7703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "faaa7365b68646198e123e92c3e81b12" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4efb3f7a9871466ca1c95f235aafdb86" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33833 } } peers { permanent_uuid: "3799df284b594f518505074efea61789" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 33151 } attrs { promote: false } } peers { permanent_uuid: "faaa7365b68646198e123e92c3e81b12" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 35317 } attrs { promote: false } } }
I20260620 20:22:58.533049 7713 sys_catalog.cc:458] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12 [sys.catalog]: This master's current role is: LEADER
I20260620 20:22:58.533125 7703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3799df284b594f518505074efea61789 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:22:58.547796 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.540674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49244 (local address 127.31.235.131:34255)
0620 20:22:58.541585 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:58.541593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:58.541615 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:22:58.542530 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:58.542536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:58.542615 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:58.543168 (+ 553us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:58.543333 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.544076 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.545728 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.546837 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.546999 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:58.547074 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:58.547333 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 20:22:58.547476 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":674,"threads_started":1}
I20260620 20:22:58.547837 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.540562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:34255 (local address 127.0.0.1:49244)
0620 20:22:58.540806 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.540827 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:58.540927 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:58.542767 (+ 1840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:58.542771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:58.542787 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:58.543019 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:58.543032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.544232 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.544237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:58.545477 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:58.545540 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.546985 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.546991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:58.547181 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:58.547207 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 20:22:58.547310 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 20:22:58.547507 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":51}
I20260620 20:22:58.549448 7563 tablet_service.cc:1467] Processing CreateTablet for tablet 78690fe784f840a98ddde29a36b25f13 (DEFAULT_TABLE table=test-table [id=a256713d08d645f28b9d036a0b2f3c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:58.549932 7563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78690fe784f840a98ddde29a36b25f13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:58.551018 7707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.541100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:39559 (local address 127.0.0.1:50206)
0620 20:22:58.541434 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.541450 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:58.541954 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:58.542737 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:58.542745 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:58.542772 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:58.543078 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:58.543090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.544852 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.544861 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:58.547635 (+ 2774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:58.547653 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.549829 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.549839 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:58.550086 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:58.550140 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 20:22:58.550217 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 20:22:58.550411 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":24}
I20260620 20:22:58.552194 7723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.541192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50206 (local address 127.31.235.130:39559)
0620 20:22:58.542359 (+ 1167us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:58.542368 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:58.542391 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:22:58.542474 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:58.542482 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:58.542579 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:58.542723 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:58.543220 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.544645 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.547910 (+ 3265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.549541 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.551445 (+ 1904us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:58.551553 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:58.551615 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 20:22:58.551810 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1015,"thread_start_us":929,"threads_started":1}
I20260620 20:22:58.552875 7422 tablet_service.cc:1467] Processing CreateTablet for tablet 78690fe784f840a98ddde29a36b25f13 (DEFAULT_TABLE table=test-table [id=a256713d08d645f28b9d036a0b2f3c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:58.553185 7422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78690fe784f840a98ddde29a36b25f13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:58.555645 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.541719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:37795 (local address 127.0.0.1:44258)
0620 20:22:58.548169 (+ 6450us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.548184 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:58.548279 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:58.548674 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:58.548678 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:58.548709 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:58.548889 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:58.548901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.551329 (+ 2428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.551334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:58.553216 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:58.553231 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.554884 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.554891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:58.555102 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:58.555137 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 20:22:58.555251 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 20:22:58.555381 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6345,"thread_start_us":10838,"threads_started":1}
I20260620 20:22:58.555768 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.541870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44258 (local address 127.31.235.129:37795)
0620 20:22:58.542569 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:58.542575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:58.548426 (+ 5851us) server_negotiation.cc:407] Connection header received
0620 20:22:58.548490 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:58.548494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:58.548552 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:58.548665 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:58.549051 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.551159 (+ 2108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.553467 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.554715 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.554881 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:58.554949 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:58.555291 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 20:22:58.555433 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":480,"threads_started":1}
I20260620 20:22:58.556522 7282 tablet_service.cc:1467] Processing CreateTablet for tablet 78690fe784f840a98ddde29a36b25f13 (DEFAULT_TABLE table=test-table [id=a256713d08d645f28b9d036a0b2f3c70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:22:58.556932 7282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78690fe784f840a98ddde29a36b25f13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:22:58.560133 7727 tablet_bootstrap.cc:492] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5: Bootstrap starting.
I20260620 20:22:58.562167 7726 tablet_bootstrap.cc:492] T 78690fe784f840a98ddde29a36b25f13 P e6c9422c263247ba8aae81e971905dc8: Bootstrap starting.
I20260620 20:22:58.562424 7727 tablet_bootstrap.cc:654] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:58.564132 7726 tablet_bootstrap.cc:654] T 78690fe784f840a98ddde29a36b25f13 P e6c9422c263247ba8aae81e971905dc8: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:58.565670 7727 tablet_bootstrap.cc:492] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5: No bootstrap required, opened a new log
I20260620 20:22:58.565965 7727 ts_tablet_manager.cc:1397] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 20:22:58.567000 7727 raft_consensus.cc:348] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 } } peers { permanent_uuid: "b23046502b304dbc98a013e3192761b5" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39559 } } peers { permanent_uuid: "838a74503a5f46b39c7553da21104e3d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37795 } }
I20260620 20:22:58.567305 7727 raft_consensus.cc:374] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:58.567431 7727 raft_consensus.cc:729] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23046502b304dbc98a013e3192761b5, State: Initialized, Role: FOLLOWER
I20260620 20:22:58.567739 7727 consensus_queue.cc:260] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 } } peers { permanent_uuid: "b23046502b304dbc98a013e3192761b5" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39559 } } peers { permanent_uuid: "838a74503a5f46b39c7553da21104e3d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37795 } }
I20260620 20:22:58.568769 7727 ts_tablet_manager.cc:1428] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 20:22:58.569525 7728 tablet_bootstrap.cc:492] T 78690fe784f840a98ddde29a36b25f13 P 838a74503a5f46b39c7553da21104e3d: Bootstrap starting.
I20260620 20:22:58.570699 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.569458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.130:39355)
0620 20:22:58.570209 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.570486 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":434,"threads_started":1}
I20260620 20:22:58.572211 7728 tablet_bootstrap.cc:654] T 78690fe784f840a98ddde29a36b25f13 P 838a74503a5f46b39c7553da21104e3d: Neither blocks nor log segments found. Creating new log.
I20260620 20:22:58.572495 7726 tablet_bootstrap.cc:492] T 78690fe784f840a98ddde29a36b25f13 P e6c9422c263247ba8aae81e971905dc8: No bootstrap required, opened a new log
I20260620 20:22:58.572700 7726 ts_tablet_manager.cc:1397] T 78690fe784f840a98ddde29a36b25f13 P e6c9422c263247ba8aae81e971905dc8: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 20:22:58.573035 7487 heartbeater.cc:503] Master 127.31.235.188:35317 was elected leader, sending a full tablet report...
I20260620 20:22:58.573737 7726 raft_consensus.cc:348] T 78690fe784f840a98ddde29a36b25f13 P e6c9422c263247ba8aae81e971905dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 } } peers { permanent_uuid: "b23046502b304dbc98a013e3192761b5" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39559 } } peers { permanent_uuid: "838a74503a5f46b39c7553da21104e3d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37795 } }
I20260620 20:22:58.574018 7726 raft_consensus.cc:374] T 78690fe784f840a98ddde29a36b25f13 P e6c9422c263247ba8aae81e971905dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:58.574111 7726 raft_consensus.cc:729] T 78690fe784f840a98ddde29a36b25f13 P e6c9422c263247ba8aae81e971905dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6c9422c263247ba8aae81e971905dc8, State: Initialized, Role: FOLLOWER
I20260620 20:22:58.574296 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.573861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.130:40177)
0620 20:22:58.574056 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.574140 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 20:22:58.574378 7726 consensus_queue.cc:260] T 78690fe784f840a98ddde29a36b25f13 P e6c9422c263247ba8aae81e971905dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 } } peers { permanent_uuid: "b23046502b304dbc98a013e3192761b5" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39559 } } peers { permanent_uuid: "838a74503a5f46b39c7553da21104e3d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37795 } }
I20260620 20:22:58.575331 7728 tablet_bootstrap.cc:492] T 78690fe784f840a98ddde29a36b25f13 P 838a74503a5f46b39c7553da21104e3d: No bootstrap required, opened a new log
I20260620 20:22:58.575500 7728 ts_tablet_manager.cc:1397] T 78690fe784f840a98ddde29a36b25f13 P 838a74503a5f46b39c7553da21104e3d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260620 20:22:58.576262 7728 raft_consensus.cc:348] T 78690fe784f840a98ddde29a36b25f13 P 838a74503a5f46b39c7553da21104e3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 } } peers { permanent_uuid: "b23046502b304dbc98a013e3192761b5" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39559 } } peers { permanent_uuid: "838a74503a5f46b39c7553da21104e3d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37795 } }
I20260620 20:22:58.576478 7728 raft_consensus.cc:374] T 78690fe784f840a98ddde29a36b25f13 P 838a74503a5f46b39c7553da21104e3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:22:58.576596 7728 raft_consensus.cc:729] T 78690fe784f840a98ddde29a36b25f13 P 838a74503a5f46b39c7553da21104e3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 838a74503a5f46b39c7553da21104e3d, State: Initialized, Role: FOLLOWER
I20260620 20:22:58.576831 7728 consensus_queue.cc:260] T 78690fe784f840a98ddde29a36b25f13 P 838a74503a5f46b39c7553da21104e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 } } peers { permanent_uuid: "b23046502b304dbc98a013e3192761b5" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39559 } } peers { permanent_uuid: "838a74503a5f46b39c7553da21104e3d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37795 } }
I20260620 20:22:58.577675 7728 ts_tablet_manager.cc:1428] T 78690fe784f840a98ddde29a36b25f13 P 838a74503a5f46b39c7553da21104e3d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 20:22:58.578734 7726 ts_tablet_manager.cc:1428] T 78690fe784f840a98ddde29a36b25f13 P e6c9422c263247ba8aae81e971905dc8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 20:22:58.586314 7628 heartbeater.cc:503] Master 127.31.235.188:35317 was elected leader, sending a full tablet report...
I20260620 20:22:58.587183 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.579368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.131:38797)
0620 20:22:58.586751 (+ 7383us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.586924 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6598,"thread_start_us":6453,"threads_started":1}
I20260620 20:22:58.588340 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.587959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.131:50833)
0620 20:22:58.588109 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.588195 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 20:22:58.589165 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.578305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.129:44341)
0620 20:22:58.588690 (+ 10385us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.588868 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10284,"thread_start_us":10021,"threads_started":1}
I20260620 20:22:58.590134 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.589755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.129:40327)
0620 20:22:58.589901 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.590000 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 20:22:58.590282 7347 heartbeater.cc:503] Master 127.31.235.188:35317 was elected leader, sending a full tablet report...
I20260620 20:22:58.849246 7731 raft_consensus.cc:482] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:22:58.849467 7731 raft_consensus.cc:504] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 } } peers { permanent_uuid: "b23046502b304dbc98a013e3192761b5" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39559 } } peers { permanent_uuid: "838a74503a5f46b39c7553da21104e3d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37795 } }
I20260620 20:22:58.851120 7731 leader_election.cc:288] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6c9422c263247ba8aae81e971905dc8 (127.31.235.131:34255), 838a74503a5f46b39c7553da21104e3d (127.31.235.129:37795)
I20260620 20:22:58.858999 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.851551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:55707 (local address 127.31.235.129:37795)
0620 20:22:58.851739 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:58.851746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:58.852274 (+ 528us) server_negotiation.cc:407] Connection header received
0620 20:22:58.852399 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:58.852403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:58.852456 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:58.852535 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:58.853027 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.853982 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.856450 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.857763 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.857934 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:58.858008 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:58.858607 (+ 599us) server_negotiation.cc:299] Negotiation successful
0620 20:22:58.858738 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:22:58.858992 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.851426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:51735 (local address 127.31.235.131:34255)
0620 20:22:58.851786 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:58.851795 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:58.851824 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:22:58.852005 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:58.852010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:58.852096 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:58.852180 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:22:58.852702 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.853769 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.856450 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.857787 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.857934 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:58.858007 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:58.858607 (+ 600us) server_negotiation.cc:299] Negotiation successful
0620 20:22:58.858738 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260620 20:22:58.859213 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.851416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:34255 (local address 127.31.235.130:51735)
0620 20:22:58.851607 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.851628 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:58.851833 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:58.852234 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:58.852241 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:58.852268 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:58.852559 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:58.852567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.853894 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.853897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:58.856292 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:58.856306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.858046 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.858059 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:58.858320 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:58.858455 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 20:22:58.858556 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 20:22:58.858753 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":226}
I20260620 20:22:58.859233 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.851434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:37795 (local address 127.31.235.130:55707)
0620 20:22:58.852133 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.852145 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 20:22:58.852288 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:22:58.852568 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:22:58.852573 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:22:58.852595 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:22:58.852884 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:58.852891 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.854105 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.854110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:58.856294 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:22:58.856306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.858082 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.858088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:22:58.858317 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:58.858455 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 20:22:58.858556 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 20:22:58.858794 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":196,"thread_start_us":353,"threads_started":1}
I20260620 20:22:58.859908 7583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78690fe784f840a98ddde29a36b25f13" candidate_uuid: "b23046502b304dbc98a013e3192761b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c9422c263247ba8aae81e971905dc8" is_pre_election: true
I20260620 20:22:58.860178 7583 raft_consensus.cc:2454] T 78690fe784f840a98ddde29a36b25f13 P e6c9422c263247ba8aae81e971905dc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b23046502b304dbc98a013e3192761b5 in term 0.
I20260620 20:22:58.860329 7302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78690fe784f840a98ddde29a36b25f13" candidate_uuid: "b23046502b304dbc98a013e3192761b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "838a74503a5f46b39c7553da21104e3d" is_pre_election: true
I20260620 20:22:58.860742 7302 raft_consensus.cc:2454] T 78690fe784f840a98ddde29a36b25f13 P 838a74503a5f46b39c7553da21104e3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b23046502b304dbc98a013e3192761b5 in term 0.
I20260620 20:22:58.860734 7377 leader_election.cc:302] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b23046502b304dbc98a013e3192761b5, e6c9422c263247ba8aae81e971905dc8; no voters:
I20260620 20:22:58.861119 7731 raft_consensus.cc:2781] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:22:58.861260 7731 raft_consensus.cc:482] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:22:58.861362 7731 raft_consensus.cc:3037] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:58.863608 7731 raft_consensus.cc:504] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 } } peers { permanent_uuid: "b23046502b304dbc98a013e3192761b5" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39559 } } peers { permanent_uuid: "838a74503a5f46b39c7553da21104e3d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37795 } }
I20260620 20:22:58.864267 7731 leader_election.cc:288] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 [CANDIDATE]: Term 1 election: Requested vote from peers e6c9422c263247ba8aae81e971905dc8 (127.31.235.131:34255), 838a74503a5f46b39c7553da21104e3d (127.31.235.129:37795)
I20260620 20:22:58.864787 7302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78690fe784f840a98ddde29a36b25f13" candidate_uuid: "b23046502b304dbc98a013e3192761b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "838a74503a5f46b39c7553da21104e3d"
I20260620 20:22:58.864787 7583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78690fe784f840a98ddde29a36b25f13" candidate_uuid: "b23046502b304dbc98a013e3192761b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6c9422c263247ba8aae81e971905dc8"
I20260620 20:22:58.864998 7302 raft_consensus.cc:3037] T 78690fe784f840a98ddde29a36b25f13 P 838a74503a5f46b39c7553da21104e3d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:58.864998 7583 raft_consensus.cc:3037] T 78690fe784f840a98ddde29a36b25f13 P e6c9422c263247ba8aae81e971905dc8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:22:58.868032 7583 raft_consensus.cc:2454] T 78690fe784f840a98ddde29a36b25f13 P e6c9422c263247ba8aae81e971905dc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b23046502b304dbc98a013e3192761b5 in term 1.
I20260620 20:22:58.868036 7302 raft_consensus.cc:2454] T 78690fe784f840a98ddde29a36b25f13 P 838a74503a5f46b39c7553da21104e3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b23046502b304dbc98a013e3192761b5 in term 1.
I20260620 20:22:58.868423 7377 leader_election.cc:302] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b23046502b304dbc98a013e3192761b5, e6c9422c263247ba8aae81e971905dc8; no voters:
I20260620 20:22:58.868793 7731 raft_consensus.cc:2781] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:22:58.869221 7731 raft_consensus.cc:686] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 [term 1 LEADER]: Becoming Leader. State: Replica: b23046502b304dbc98a013e3192761b5, State: Running, Role: LEADER
I20260620 20:22:58.869920 7731 consensus_queue.cc:237] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 } } peers { permanent_uuid: "b23046502b304dbc98a013e3192761b5" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39559 } } peers { permanent_uuid: "838a74503a5f46b39c7553da21104e3d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37795 } }
I20260620 20:22:58.870353 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.869915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.130:56355)
0620 20:22:58.870089 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.870207 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 20:22:58.873692 7161 catalog_manager.cc:5495] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b23046502b304dbc98a013e3192761b5 (127.31.235.130). New cstate: current_term: 1 leader_uuid: "b23046502b304dbc98a013e3192761b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b23046502b304dbc98a013e3192761b5" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "838a74503a5f46b39c7553da21104e3d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37795 } health_report { overall_health: UNKNOWN } } }
I20260620 20:22:58.918571 7723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.913600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50222 (local address 127.31.235.130:39559)
0620 20:22:58.913812 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:58.913816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:58.913836 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:22:58.914085 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:58.914089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:58.914172 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:58.914301 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:22:58.914597 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.915315 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.916578 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.916971 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.917070 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:58.917421 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:22:58.918025 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:22:58.918172 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:58.918287 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 20:22:58.918363 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:22:58.922359 7583 raft_consensus.cc:1261] T 78690fe784f840a98ddde29a36b25f13 P e6c9422c263247ba8aae81e971905dc8 [term 1 FOLLOWER]: Refusing update from remote peer b23046502b304dbc98a013e3192761b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:58.922624 7302 raft_consensus.cc:1261] T 78690fe784f840a98ddde29a36b25f13 P 838a74503a5f46b39c7553da21104e3d [term 1 FOLLOWER]: Refusing update from remote peer b23046502b304dbc98a013e3192761b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:22:58.923077 7739 consensus_queue.cc:1035] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6c9422c263247ba8aae81e971905dc8" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 34255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:58.923427 7731 consensus_queue.cc:1035] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "838a74503a5f46b39c7553da21104e3d" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:22:58.925817 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.925286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.131:37577)
0620 20:22:58.925472 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.925621 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 20:22:58.927099 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.926538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.129:49763)
0620 20:22:58.926732 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.926874 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 20:22:58.939824 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.939362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.129:49273)
0620 20:22:58.939534 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.939623 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 20:22:58.945633 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.944867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.130:33003)
0620 20:22:58.945316 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.945411 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310}
I20260620 20:22:58.947409 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.946956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.130:37303)
0620 20:22:58.947147 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.947253 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 20:22:58.949400 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.948943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.31.235.131:49651)
0620 20:22:58.949152 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:58.949235 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 20:22:58.952297 7162 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:58.952538 7162 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:22:58.958909 7162 catalog_manager.cc:5782] T 00000000000000000000000000000000 P faaa7365b68646198e123e92c3e81b12: Sending DeleteTablet for 3 replicas of tablet 78690fe784f840a98ddde29a36b25f13
I20260620 20:22:58.960220 7563 tablet_service.cc:1514] Processing DeleteTablet for tablet 78690fe784f840a98ddde29a36b25f13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:58 UTC) from {username='oryx', principal='oryx/127.31.235.188@KRBTEST.COM'} at 127.0.0.1:49244
I20260620 20:22:58.960253 7422 tablet_service.cc:1514] Processing DeleteTablet for tablet 78690fe784f840a98ddde29a36b25f13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:58 UTC) from {username='oryx', principal='oryx/127.31.235.188@KRBTEST.COM'} at 127.0.0.1:50206
I20260620 20:22:58.961278 7282 tablet_service.cc:1514] Processing DeleteTablet for tablet 78690fe784f840a98ddde29a36b25f13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:22:58 UTC) from {username='oryx', principal='oryx/127.31.235.188@KRBTEST.COM'} at 127.0.0.1:44258
I20260620 20:22:58.962112 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 7139
I20260620 20:22:58.983162 7753 tablet_replica.cc:330] T 78690fe784f840a98ddde29a36b25f13 P 838a74503a5f46b39c7553da21104e3d: stopping tablet replica
I20260620 20:22:58.983387 7753 raft_consensus.cc:2229] T 78690fe784f840a98ddde29a36b25f13 P 838a74503a5f46b39c7553da21104e3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:22:58.983479 7751 tablet_replica.cc:330] T 78690fe784f840a98ddde29a36b25f13 P e6c9422c263247ba8aae81e971905dc8: stopping tablet replica
I20260620 20:22:58.983731 7753 raft_consensus.cc:2258] T 78690fe784f840a98ddde29a36b25f13 P 838a74503a5f46b39c7553da21104e3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:58.983773 7751 raft_consensus.cc:2229] T 78690fe784f840a98ddde29a36b25f13 P e6c9422c263247ba8aae81e971905dc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:22:58.984566 7751 raft_consensus.cc:2258] T 78690fe784f840a98ddde29a36b25f13 P e6c9422c263247ba8aae81e971905dc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:58.986073 7753 ts_tablet_manager.cc:1905] T 78690fe784f840a98ddde29a36b25f13 P 838a74503a5f46b39c7553da21104e3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:58.991928 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:58.963521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44266 (local address 127.31.235.129:37795)
0620 20:22:58.963677 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:58.963683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:58.973574 (+ 9891us) server_negotiation.cc:407] Connection header received
0620 20:22:58.974070 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:58.974077 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:58.974180 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:58.974263 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:58.976276 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.977471 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.979285 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:58.979913 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:58.979999 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:58.983806 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:58.983823 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:58.983844 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:58.983884 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:58.988456 (+ 4572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:58.989282 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:58.989292 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:58.989296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:58.989474 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:58.989949 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:58.989955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:58.989958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:58.990446 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:58.990756 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:58.991315 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 20:22:58.991550 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:22:58.993593 7342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:58.994264 7751 ts_tablet_manager.cc:1905] T 78690fe784f840a98ddde29a36b25f13 P e6c9422c263247ba8aae81e971905dc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:59.000058 7752 tablet_replica.cc:330] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5: stopping tablet replica
I20260620 20:22:59.000375 7752 raft_consensus.cc:2229] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:22:59.001161 7752 raft_consensus.cc:2258] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:22:59.002460 7752 ts_tablet_manager.cc:1905] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:22:59.004027 7751 ts_tablet_manager.cc:1918] T 78690fe784f840a98ddde29a36b25f13 P e6c9422c263247ba8aae81e971905dc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:22:59.004173 7751 log.cc:1196] T 78690fe784f840a98ddde29a36b25f13 P e6c9422c263247ba8aae81e971905dc8: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/78690fe784f840a98ddde29a36b25f13
I20260620 20:22:59.004616 7751 ts_tablet_manager.cc:1939] T 78690fe784f840a98ddde29a36b25f13 P e6c9422c263247ba8aae81e971905dc8: Deleting consensus metadata
I20260620 20:22:59.006019 7148 catalog_manager.cc:4841] TS e6c9422c263247ba8aae81e971905dc8 (127.31.235.131:34255): tablet 78690fe784f840a98ddde29a36b25f13 (table test-table [id=a256713d08d645f28b9d036a0b2f3c70]) successfully deleted
I20260620 20:22:59.010900 7752 ts_tablet_manager.cc:1918] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:22:59.011053 7752 log.cc:1196] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/78690fe784f840a98ddde29a36b25f13
I20260620 20:22:59.011551 7752 ts_tablet_manager.cc:1939] T 78690fe784f840a98ddde29a36b25f13 P b23046502b304dbc98a013e3192761b5: Deleting consensus metadata
I20260620 20:22:59.012790 7148 catalog_manager.cc:4841] TS b23046502b304dbc98a013e3192761b5 (127.31.235.130:39559): tablet 78690fe784f840a98ddde29a36b25f13 (table test-table [id=a256713d08d645f28b9d036a0b2f3c70]) successfully deleted
I20260620 20:22:59.130296 7753 ts_tablet_manager.cc:1918] T 78690fe784f840a98ddde29a36b25f13 P 838a74503a5f46b39c7553da21104e3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:22:59.130520 7753 log.cc:1196] T 78690fe784f840a98ddde29a36b25f13 P 838a74503a5f46b39c7553da21104e3d: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/78690fe784f840a98ddde29a36b25f13
I20260620 20:22:59.130904 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 7139
I20260620 20:22:59.151329 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:59.150388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.0.0.1:57112)
0620 20:22:59.150937 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:59.151065 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":303,"threads_started":1}
I20260620 20:22:59.154592 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 7359
I20260620 20:22:59.154706 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:59.154259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33833 (local address 127.0.0.1:57122)
0620 20:22:59.154443 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:59.154530 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:33833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 20:22:59.155246 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:59.154739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:37795 (local address 127.0.0.1:44268)
0620 20:22:59.154979 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:59.155064 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"threads_started":1}
W20260620 20:22:59.155827 7146 catalog_manager.cc:4584] TS 838a74503a5f46b39c7553da21104e3d (127.31.235.129:37795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78690fe784f840a98ddde29a36b25f13: Network error: Client connection negotiation failed: client connection to 127.31.235.129:37795: connect: Connection refused (error 111)
I20260620 20:22:59.168826 7723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:59.155544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.31.235.130:39559)
0620 20:22:59.155711 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:59.155721 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:59.155737 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:22:59.155789 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:59.155793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:59.155855 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:59.155979 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:59.157360 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:59.158323 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:59.159674 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:59.160119 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:59.160269 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:59.162759 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:59.162775 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:59.162790 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:59.162828 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:59.166124 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:59.166794 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:59.166803 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:59.166807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:59.166969 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:59.167338 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:59.167344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:59.167347 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:59.167680 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:59.168054 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:59.168289 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 20:22:59.168500 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:22:59.169471 7482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:59.208680 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:59.207991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:37795 (local address 127.0.0.1:44270)
0620 20:22:59.208307 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:59.208469 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260620 20:22:59.209829 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:59.209428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:37795 (local address 127.0.0.1:44276)
0620 20:22:59.209613 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:59.209697 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 20:22:59.279492 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:59.278933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:37795 (local address 127.0.0.1:44280)
0620 20:22:59.279189 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:59.279293 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 20:22:59.280560 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:59.280166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:37795 (local address 127.0.0.1:44294)
0620 20:22:59.280333 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:59.280415 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 20:22:59.288506 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 7359
I20260620 20:22:59.307971 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 7500
I20260620 20:22:59.321688 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:59.308810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49250 (local address 127.31.235.131:34255)
0620 20:22:59.308986 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 20:22:59.308993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:22:59.309056 (+ 63us) server_negotiation.cc:407] Connection header received
0620 20:22:59.309299 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:22:59.309303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:22:59.309365 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:22:59.309484 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:22:59.311033 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:59.311762 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:59.312964 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:22:59.313369 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:22:59.313504 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:22:59.316090 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:22:59.316104 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:22:59.316121 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:22:59.316152 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:22:59.318982 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:59.319656 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:59.319663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:59.319666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:59.319790 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:22:59.320180 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:22:59.320190 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:22:59.320192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:22:59.320540 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:22:59.320853 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 20:22:59.321132 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 20:22:59.321342 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 20:22:59.322373 7623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:59.387979 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:59.387312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:37795 (local address 127.0.0.1:44304)
0620 20:22:59.387578 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:59.387753 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 20:22:59.388974 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:59.388683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:37795 (local address 127.0.0.1:44306)
0620 20:22:59.388804 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:59.388881 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 20:22:59.420750 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 7500
I20260620 20:22:59.437902 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 6827
I20260620 20:22:59.439148 6994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:59.556669 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 6827
I20260620 20:22:59.567929 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:59.567349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:37795 (local address 127.0.0.1:44312)
0620 20:22:59.567603 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:59.567720 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 20:22:59.569096 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:22:59.568656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:37795 (local address 127.0.0.1:44316)
0620 20:22:59.568829 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 20:22:59.568929 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:37795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 20:22:59.577937 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 6928
I20260620 20:22:59.578915 7192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:22:59.698175 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 6928
2026-06-20T20:22:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9879 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:22:59 dist-test-slave-zzwf krb5kdc[7765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:22:59 dist-test-slave-zzwf krb5kdc[7765](info): set up 2 sockets
Jun 20 20:22:59 dist-test-slave-zzwf krb5kdc[7765](info): commencing operation
krb5kdc: starting...
W20260620 20:23:00.798939 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:23:00 dist-test-slave-zzwf krb5kdc[7765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:23:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:23:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.190.keytab.
Entry for principal oryx/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.190.keytab.
I20260620 20:23:01.017958 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:45937
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.190.keytab
--principal=oryx/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:37981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:45937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:23:01.349642 7778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:01.349984 7778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:01.350131 7778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:01.357306 7778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:23:01.357450 7778 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 20:23:01.357571 7778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:01.357693 7778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:01.357782 7778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:23:01.357872 7778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:23:01.367802 7778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:37981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:45937
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.190.keytab
--principal=oryx/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:45937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:01.369086 7778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:01.370464 7778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:01.382296 7783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:01.382297 7784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:01.382337 7786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:01.383247 7778 server_base.cc:1034] running on GCE node
I20260620 20:23:01.383982 7778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:01.385258 7778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:01.386502 7778 hybrid_clock.cc:648] HybridClock initialized: now 1781986981386433 us; error 33 us; skew 500 ppm
I20260620 20:23:01.386788 7778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:23:01 dist-test-slave-zzwf krb5kdc[7765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986981, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:23:01.393424 7778 init.cc:377] Logged in from keytab as oryx/127.31.235.190@KRBTEST.COM (short username oryx)
I20260620 20:23:01.395327 7778 webserver.cc:460] Webserver started at http://127.31.235.190:43739/ using document root <none> and password file <none>
I20260620 20:23:01.396004 7778 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:01.396090 7778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:01.396363 7778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:23:01.399307 7778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "01679f23407f4a689678b3c4bd7ba0f5"
format_stamp: "Formatted at 2026-06-20 20:23:01 on dist-test-slave-zzwf"
server_key: "06a4e464c66af332ac6b2633ca8ad889"
server_key_iv: "69be0c11ba73db09fabacfca003a30d8"
server_key_version: "encryptionkey@0"
I20260620 20:23:01.400015 7778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "01679f23407f4a689678b3c4bd7ba0f5"
format_stamp: "Formatted at 2026-06-20 20:23:01 on dist-test-slave-zzwf"
server_key: "06a4e464c66af332ac6b2633ca8ad889"
server_key_iv: "69be0c11ba73db09fabacfca003a30d8"
server_key_version: "encryptionkey@0"
I20260620 20:23:01.404873 7778 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 20:23:01.408232 7793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:01.409662 7778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:23:01.409832 7778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "01679f23407f4a689678b3c4bd7ba0f5"
format_stamp: "Formatted at 2026-06-20 20:23:01 on dist-test-slave-zzwf"
server_key: "06a4e464c66af332ac6b2633ca8ad889"
server_key_iv: "69be0c11ba73db09fabacfca003a30d8"
server_key_version: "encryptionkey@0"
I20260620 20:23:01.410044 7778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:01.426234 7778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:01.430223 7778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:01.430500 7778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:01.452517 7778 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:45937
I20260620 20:23:01.452572 7854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:45937 every 8 connection(s)
I20260620 20:23:01.453878 7778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:23:01.455245 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 7778
I20260620 20:23:01.455463 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:23:01.456358 32686 external_mini_cluster.cc:1428] Setting key 2c8ece4eec40d91886410c19e0a0f2a3
I20260620 20:23:01.459017 7855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 20:23:01 dist-test-slave-zzwf krb5kdc[7765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM
I20260620 20:23:01.473155 7855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5: Bootstrap starting.
I20260620 20:23:01.476529 7855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:01.477888 7855 log.cc:824] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5: Log is configured to *not* fsync() on all Append() calls
I20260620 20:23:01.479965 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:01.458558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58056 (local address 127.31.235.190:45937)
0620 20:23:01.459357 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:01.459368 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:01.459407 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:23:01.460959 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:01.460979 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:01.461359 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:01.461760 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:01.463617 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:01.464984 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:01.466919 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:01.467595 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:01.467732 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:01.472649 (+ 4917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:01.472657 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:01.472672 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:01.472711 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:01.475953 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:01.476660 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:01.476670 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:01.476677 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:01.476854 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:01.477311 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:01.477315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:01.477319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:01.477879 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:01.478182 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:01.478622 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 20:23:01.479033 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":375,"threads_started":1}
I20260620 20:23:01.481046 7855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5: No bootstrap required, opened a new log
I20260620 20:23:01.488814 7855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01679f23407f4a689678b3c4bd7ba0f5" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 45937 } }
I20260620 20:23:01.489215 7855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:01.489334 7855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01679f23407f4a689678b3c4bd7ba0f5, State: Initialized, Role: FOLLOWER
I20260620 20:23:01.489969 7855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01679f23407f4a689678b3c4bd7ba0f5" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 45937 } }
I20260620 20:23:01.490224 7855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:23:01.490378 7855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:23:01.490573 7855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:01.492856 7855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01679f23407f4a689678b3c4bd7ba0f5" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 45937 } }
I20260620 20:23:01.493427 7855 leader_election.cc:302] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01679f23407f4a689678b3c4bd7ba0f5; no voters:
I20260620 20:23:01.494203 7855 leader_election.cc:288] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:23:01.494324 7860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:23:01.495074 7860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5 [term 1 LEADER]: Becoming Leader. State: Replica: 01679f23407f4a689678b3c4bd7ba0f5, State: Running, Role: LEADER
I20260620 20:23:01.495569 7860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01679f23407f4a689678b3c4bd7ba0f5" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 45937 } }
I20260620 20:23:01.496376 7855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:23:01.498435 7861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01679f23407f4a689678b3c4bd7ba0f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01679f23407f4a689678b3c4bd7ba0f5" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 45937 } } }
I20260620 20:23:01.499430 7861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5 [sys.catalog]: This master's current role is: LEADER
I20260620 20:23:01.501945 7862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01679f23407f4a689678b3c4bd7ba0f5. Latest consensus state: current_term: 1 leader_uuid: "01679f23407f4a689678b3c4bd7ba0f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01679f23407f4a689678b3c4bd7ba0f5" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 45937 } } }
I20260620 20:23:01.502431 7862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5 [sys.catalog]: This master's current role is: LEADER
I20260620 20:23:01.503654 7869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:23:01.508672 7869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:23:01.513979 7877 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 20:23:01.518744 7869 catalog_manager.cc:1347] Generated new cluster ID: be86ce887c3d4783b2df5eae1186bd4d
I20260620 20:23:01.518867 7869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:23:01.543562 7869 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:23:01.544975 7869 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:23:01.554307 7869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5: Generated new TSK 0
I20260620 20:23:01.555132 7869 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.129.keytab.
Entry for principal oryx/127.31.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.129.keytab.
I20260620 20:23:01.613459 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:45937
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.129.keytab
--principal=oryx/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:37981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:23:01.960664 7881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:01.960976 7881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:01.961074 7881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:01.968346 7881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:01.968468 7881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:01.968552 7881 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:23:01.968631 7881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:23:01.979892 7881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:37981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.129.keytab
--principal=oryx/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.235.190:45937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:01.981298 7881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:01.982774 7881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:01.997344 7886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:01.997336 7889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:01.997268 7887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:01.998268 7881 server_base.cc:1034] running on GCE node
I20260620 20:23:01.999307 7881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:02.000721 7881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:02.002007 7881 hybrid_clock.cc:648] HybridClock initialized: now 1781986982001898 us; error 80 us; skew 500 ppm
I20260620 20:23:02.002408 7881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:23:01 dist-test-slave-zzwf krb5kdc[7765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986981, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:23:02.005839 7881 init.cc:377] Logged in from keytab as oryx/127.31.235.129@KRBTEST.COM (short username oryx)
I20260620 20:23:02.007371 7881 webserver.cc:460] Webserver started at http://127.31.235.129:37693/ using document root <none> and password file <none>
I20260620 20:23:02.008077 7881 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:02.008167 7881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:02.008430 7881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:23:02.011399 7881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "db89954f95d646f48ec84241c5307e2e"
format_stamp: "Formatted at 2026-06-20 20:23:02 on dist-test-slave-zzwf"
server_key: "68cdb2e9e6d7bd452af7b2c98a4b7381"
server_key_iv: "7af100be009a788162b632d1f3139af8"
server_key_version: "encryptionkey@0"
I20260620 20:23:02.012130 7881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "db89954f95d646f48ec84241c5307e2e"
format_stamp: "Formatted at 2026-06-20 20:23:02 on dist-test-slave-zzwf"
server_key: "68cdb2e9e6d7bd452af7b2c98a4b7381"
server_key_iv: "7af100be009a788162b632d1f3139af8"
server_key_version: "encryptionkey@0"
I20260620 20:23:02.017426 7881 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 20:23:02.020850 7896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:02.022364 7881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:23:02.022563 7881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "db89954f95d646f48ec84241c5307e2e"
format_stamp: "Formatted at 2026-06-20 20:23:02 on dist-test-slave-zzwf"
server_key: "68cdb2e9e6d7bd452af7b2c98a4b7381"
server_key_iv: "7af100be009a788162b632d1f3139af8"
server_key_version: "encryptionkey@0"
I20260620 20:23:02.022809 7881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:02.035611 7881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:02.040503 7881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:02.040824 7881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:02.043792 7881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:23:02.043917 7881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:02.044067 7881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:23:02.044209 7881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 20 20:23:02 dist-test-slave-zzwf krb5kdc[7765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986981, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM
I20260620 20:23:02.069839 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:02.044358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:51455 (local address 127.31.235.190:45937)
0620 20:23:02.047179 (+ 2821us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:02.047186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:02.047212 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:23:02.048814 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:02.048825 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:02.048934 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:02.049075 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:02.051255 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:02.052631 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:02.055426 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:02.056259 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:02.056525 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:02.062332 (+ 5807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:02.062339 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:02.062344 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:02.062397 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:02.066196 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:02.067114 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:02.067120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:02.067124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:02.067298 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:02.067842 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:02.067846 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:02.067848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:02.068275 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:02.068438 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:02.069247 (+ 809us) server_negotiation.cc:299] Negotiation successful
0620 20:23:02.069459 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2671,"thread_start_us":2545,"threads_started":1}
I20260620 20:23:02.070945 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:02.045126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:45937 (local address 127.31.235.129:51455)
0620 20:23:02.046242 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:02.046326 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:02.048498 (+ 2172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:02.049313 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:02.049320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:02.050199 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:02.051023 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:02.051067 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:02.052820 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:02.052827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:02.055271 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:02.055291 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:02.057054 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:02.057066 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:02.057519 (+ 453us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:02.058567 (+ 1048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:02.058613 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:02.061981 (+ 3368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:23:02.066385 (+ 4404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:02.066395 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:02.066425 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:02.066972 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:02.067438 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:02.067446 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:02.067450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:02.067705 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:02.068437 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:23:02.068446 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:23:02.068887 (+ 441us) client_negotiation.cc:769] Sending connection context
0620 20:23:02.069143 (+ 256us) client_negotiation.cc:240] Negotiation successful
0620 20:23:02.069709 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":460,"threads_started":1}
I20260620 20:23:02.092757 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:02.077335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:49201 (local address 127.31.235.190:45937)
0620 20:23:02.077568 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:02.077576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:02.078266 (+ 690us) server_negotiation.cc:407] Connection header received
0620 20:23:02.078544 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:02.078549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:02.078616 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:02.078704 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:02.080351 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:02.081120 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:02.084017 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:02.084470 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:02.084622 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:02.087339 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:02.087346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:02.087354 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:02.087394 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:02.090180 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:02.090905 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:02.090911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:02.090914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:02.091030 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:02.091392 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:02.091397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:02.091400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:02.091668 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:02.091815 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:02.092193 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 20:23:02.092404 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:23:02.092854 7972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:02.077310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:45937 (local address 127.31.235.129:49201)
0620 20:23:02.078104 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:02.078133 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:02.078327 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:02.078776 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:02.078784 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:02.079633 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:02.080145 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:02.080162 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:02.081282 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:02.081291 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:02.083855 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:02.083878 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:02.084692 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:02.084703 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:02.084988 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:02.085885 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:02.085916 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:02.087069 (+ 1153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:23:02.090324 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:02.090329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:02.090333 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:02.090773 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:02.091146 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:02.091152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:02.091155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:02.091283 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:02.091780 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:23:02.091785 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:23:02.091958 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 20:23:02.092090 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 20:23:02.092403 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":383,"threads_started":1}
I20260620 20:23:02.101183 7881 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:35757
I20260620 20:23:02.101287 8016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:35757 every 8 connection(s)
I20260620 20:23:02.102510 7881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:23:02.104674 8017 heartbeater.cc:347] Connected to a master server at 127.31.235.190:45937
I20260620 20:23:02.104903 8017 heartbeater.cc:464] Registering TS with master...
I20260620 20:23:02.105651 8017 heartbeater.cc:511] Master 127.31.235.190:45937 requested a full tablet report, sending...
I20260620 20:23:02.107674 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 7881
I20260620 20:23:02.107648 7810 ts_manager.cc:194] Registered new tserver with Master: db89954f95d646f48ec84241c5307e2e (127.31.235.129:35757)
I20260620 20:23:02.107872 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:23:02.108536 32686 external_mini_cluster.cc:1428] Setting key 42e798c3ccfd976f00dd98e3a06159ab
I20260620 20:23:02.109385 7810 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:51455
WARNING: no policy specified for oryx/127.31.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.130.keytab.
Entry for principal oryx/127.31.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.130.keytab.
I20260620 20:23:02.158102 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:45937
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.130.keytab
--principal=oryx/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:37981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:23:02.478557 8021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:02.478814 8021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:02.478976 8021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:02.486016 8021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:02.486124 8021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:02.486315 8021 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:23:02.486431 8021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:23:02.495767 8021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:37981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.130.keytab
--principal=oryx/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.235.190:45937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:02.496959 8021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:02.498247 8021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:02.510625 8026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:02.510658 8027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:02.510686 8029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:02.511917 8021 server_base.cc:1034] running on GCE node
I20260620 20:23:02.512513 8021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:02.513454 8021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:02.514742 8021 hybrid_clock.cc:648] HybridClock initialized: now 1781986982514660 us; error 64 us; skew 500 ppm
I20260620 20:23:02.515004 8021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:23:02 dist-test-slave-zzwf krb5kdc[7765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986982, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:23:02.517949 8021 init.cc:377] Logged in from keytab as oryx/127.31.235.130@KRBTEST.COM (short username oryx)
I20260620 20:23:02.519320 8021 webserver.cc:460] Webserver started at http://127.31.235.130:45853/ using document root <none> and password file <none>
I20260620 20:23:02.519992 8021 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:02.520073 8021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:02.520443 8021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:23:02.523347 8021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f"
format_stamp: "Formatted at 2026-06-20 20:23:02 on dist-test-slave-zzwf"
server_key: "2bb7bc8fc609e7bfc42ea6273c2c7019"
server_key_iv: "9bb66f58d56dd84b620043735bde76d3"
server_key_version: "encryptionkey@0"
I20260620 20:23:02.524127 8021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f"
format_stamp: "Formatted at 2026-06-20 20:23:02 on dist-test-slave-zzwf"
server_key: "2bb7bc8fc609e7bfc42ea6273c2c7019"
server_key_iv: "9bb66f58d56dd84b620043735bde76d3"
server_key_version: "encryptionkey@0"
I20260620 20:23:02.529402 8021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260620 20:23:02.532791 8036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:02.534144 8021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:23:02.534332 8021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f"
format_stamp: "Formatted at 2026-06-20 20:23:02 on dist-test-slave-zzwf"
server_key: "2bb7bc8fc609e7bfc42ea6273c2c7019"
server_key_iv: "9bb66f58d56dd84b620043735bde76d3"
server_key_version: "encryptionkey@0"
I20260620 20:23:02.534544 8021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:02.558290 8021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:02.562673 8021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:02.562955 8021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:02.566329 8021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:23:02.566473 8021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 20:23:02.566799 8021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:23:02.566884 8021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:23:02 dist-test-slave-zzwf krb5kdc[7765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986982, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM
I20260620 20:23:02.590828 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:02.566457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:34553 (local address 127.31.235.190:45937)
0620 20:23:02.566626 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:02.566631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:02.568150 (+ 1519us) server_negotiation.cc:407] Connection header received
0620 20:23:02.571091 (+ 2941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:02.571095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:02.571159 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:02.571291 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:02.574323 (+ 3032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:02.575676 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:02.577948 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:02.578652 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:02.578803 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:02.583332 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:02.583343 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:02.583351 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:02.583407 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:02.587070 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:02.588280 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:02.588286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:02.588291 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:02.588453 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:02.588949 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:02.588954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:02.588958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:02.589353 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:02.589562 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:02.590243 (+ 681us) server_negotiation.cc:299] Negotiation successful
0620 20:23:02.590449 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 20:23:02.591787 8045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:02.567024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:45937 (local address 127.31.235.130:34553)
0620 20:23:02.567964 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:02.568018 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:02.570778 (+ 2760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:02.571552 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:02.571570 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:02.572909 (+ 1339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:02.574065 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:02.574092 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:02.575852 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:02.575856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:02.577798 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:02.577809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:02.579013 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:02.579022 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:02.579352 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:02.580143 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:02.580192 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:02.583081 (+ 2889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:23:02.587229 (+ 4148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:02.587240 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:02.587258 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:02.588091 (+ 833us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:02.588586 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:02.588590 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:02.588593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:02.588808 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:02.589516 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:23:02.589529 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:23:02.589891 (+ 362us) client_negotiation.cc:769] Sending connection context
0620 20:23:02.590158 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 20:23:02.590674 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":409,"threads_started":1}
I20260620 20:23:02.612511 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:02.598322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:57973 (local address 127.31.235.190:45937)
0620 20:23:02.598551 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:02.598559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:02.599148 (+ 589us) server_negotiation.cc:407] Connection header received
0620 20:23:02.599341 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:02.599348 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:02.599435 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:02.599538 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:02.601136 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:02.602233 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:02.603544 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:02.603952 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:02.604093 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:02.605978 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:02.605982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:02.605986 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:02.606014 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:02.609117 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:02.609987 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:02.609995 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:02.610000 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:02.610233 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:02.610808 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:02.610816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:02.610819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:02.611291 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:02.611466 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:02.611953 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 20:23:02.612160 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:23:02.612629 8111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:02.598295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:45937 (local address 127.31.235.130:57973)
0620 20:23:02.599031 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:02.599053 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:02.599197 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:02.599680 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:02.599684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:02.600358 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:02.600991 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:02.601002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:02.602375 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:02.602381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:02.603402 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:02.603413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:02.604135 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:02.604141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:02.604306 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:02.604994 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:02.605010 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:02.605782 (+ 772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:23:02.609256 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:02.609261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:02.609265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:02.609871 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:02.610379 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:02.610386 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:02.610392 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:02.610591 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:02.611438 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:23:02.611448 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:23:02.611682 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 20:23:02.611868 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 20:23:02.612181 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":346,"threads_started":1}
I20260620 20:23:02.620649 8021 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:39743
I20260620 20:23:02.620672 8155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:39743 every 8 connection(s)
I20260620 20:23:02.621920 8021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:23:02.623862 8156 heartbeater.cc:347] Connected to a master server at 127.31.235.190:45937
I20260620 20:23:02.624090 8156 heartbeater.cc:464] Registering TS with master...
I20260620 20:23:02.624697 8156 heartbeater.cc:511] Master 127.31.235.190:45937 requested a full tablet report, sending...
I20260620 20:23:02.625978 7809 ts_manager.cc:194] Registered new tserver with Master: d0e64797c14e4e8e8f4bc6eb2463ae1f (127.31.235.130:39743)
I20260620 20:23:02.626850 7809 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:34553
I20260620 20:23:02.628492 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 8021
I20260620 20:23:02.628648 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:23:02.629051 32686 external_mini_cluster.cc:1428] Setting key 019d96a5ec23cd95ee048c0d16065a33
WARNING: no policy specified for oryx/127.31.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.131.keytab.
Entry for principal oryx/127.31.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.131.keytab.
I20260620 20:23:02.673480 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:45937
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.131.keytab
--principal=oryx/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:37981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:23:02.991592 8160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:02.991853 8160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:02.992010 8160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:02.999671 8160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:02.999773 8160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:02.999909 8160 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 20:23:03.000025 8160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:23:03.009723 8160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:37981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.131.keytab
--principal=oryx/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.235.190:45937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:03.010979 8160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:03.012276 8160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:03.025738 8168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:03.025687 8166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:03.026062 8165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:03.026381 8160 server_base.cc:1034] running on GCE node
I20260620 20:23:03.027148 8160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:03.028141 8160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:03.029421 8160 hybrid_clock.cc:648] HybridClock initialized: now 1781986983029319 us; error 70 us; skew 500 ppm
I20260620 20:23:03.029783 8160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:23:03 dist-test-slave-zzwf krb5kdc[7765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986983, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:23:03.032874 8160 init.cc:377] Logged in from keytab as oryx/127.31.235.131@KRBTEST.COM (short username oryx)
I20260620 20:23:03.034415 8160 webserver.cc:460] Webserver started at http://127.31.235.131:43391/ using document root <none> and password file <none>
I20260620 20:23:03.035095 8160 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:03.035226 8160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:03.035557 8160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:23:03.038573 8160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "df43785430814849be835b580aabb724"
format_stamp: "Formatted at 2026-06-20 20:23:03 on dist-test-slave-zzwf"
server_key: "6816b1e96b67e852e09906b547417293"
server_key_iv: "99f02d4707c6a31985fb2d45ce17eaad"
server_key_version: "encryptionkey@0"
I20260620 20:23:03.039407 8160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "df43785430814849be835b580aabb724"
format_stamp: "Formatted at 2026-06-20 20:23:03 on dist-test-slave-zzwf"
server_key: "6816b1e96b67e852e09906b547417293"
server_key_iv: "99f02d4707c6a31985fb2d45ce17eaad"
server_key_version: "encryptionkey@0"
I20260620 20:23:03.044395 8160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:23:03.047876 8175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:03.049081 8160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 20:23:03.049275 8160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "df43785430814849be835b580aabb724"
format_stamp: "Formatted at 2026-06-20 20:23:03 on dist-test-slave-zzwf"
server_key: "6816b1e96b67e852e09906b547417293"
server_key_iv: "99f02d4707c6a31985fb2d45ce17eaad"
server_key_version: "encryptionkey@0"
I20260620 20:23:03.049499 8160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:03.067587 8160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:03.071448 8160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:03.071722 8160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:03.074680 8160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:23:03.074832 8160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:03.075001 8160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:23:03.075086 8160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:23:03 dist-test-slave-zzwf krb5kdc[7765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986983, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM
I20260620 20:23:03.099647 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.075645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:49859 (local address 127.31.235.190:45937)
0620 20:23:03.076601 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:03.076610 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:03.077140 (+ 530us) server_negotiation.cc:407] Connection header received
0620 20:23:03.079198 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:03.079204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:03.079292 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:03.079401 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:03.082372 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.083404 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.085445 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.085981 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.086126 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.091594 (+ 5468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:03.091604 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:03.091611 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:03.091673 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:03.094891 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:03.096079 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:03.096087 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:03.096091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:03.096249 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:03.096947 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:03.096955 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:03.096961 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:03.097470 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:03.097695 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:03.098758 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0620 20:23:03.099075 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793}
I20260620 20:23:03.100211 8184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.076146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:45937 (local address 127.31.235.131:49859)
0620 20:23:03.076956 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:03.077003 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:03.078906 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:03.079873 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:03.079887 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:03.080961 (+ 1074us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:03.082130 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:03.082172 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.083576 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.083582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:03.085298 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:03.085311 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.086245 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.086252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:03.086601 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.087569 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:03.087609 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:03.091267 (+ 3658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:23:03.095064 (+ 3797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:03.095079 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:03.095103 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:03.095875 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:03.096365 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:03.096371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:03.096374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:03.096753 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:03.097652 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:23:03.097663 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:23:03.098119 (+ 456us) client_negotiation.cc:769] Sending connection context
0620 20:23:03.098496 (+ 377us) client_negotiation.cc:240] Negotiation successful
0620 20:23:03.098950 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":412,"threads_started":1}
I20260620 20:23:03.112377 8017 heartbeater.cc:503] Master 127.31.235.190:45937 was elected leader, sending a full tablet report...
I20260620 20:23:03.119930 8240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.106580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:45937 (local address 127.31.235.131:48013)
0620 20:23:03.107249 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:03.107265 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:03.107414 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:03.107737 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:03.107742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:03.108180 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:03.108492 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:03.108501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.109545 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.109554 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:03.111031 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:03.111042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.111790 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.111796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:03.111975 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.112764 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:03.112780 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:03.113941 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:23:03.117320 (+ 3379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:03.117327 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:03.117332 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:03.117881 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:03.118347 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:03.118350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:03.118352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:03.118464 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:03.119164 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:23:03.119167 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:23:03.119318 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 20:23:03.119417 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 20:23:03.119621 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":368,"threads_started":1}
I20260620 20:23:03.120272 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.106538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:48013 (local address 127.31.235.190:45937)
0620 20:23:03.106730 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:03.106737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:03.107348 (+ 611us) server_negotiation.cc:407] Connection header received
0620 20:23:03.107540 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:03.107546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:03.107609 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:03.107707 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:03.108632 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.109394 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.111173 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.111604 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.111738 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.114104 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:03.114109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:03.114112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:03.114138 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:03.117129 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:03.118009 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:03.118018 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:03.118025 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:03.118221 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:03.118581 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:03.118598 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:03.118603 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:03.119049 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:03.119242 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:03.119577 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 20:23:03.119871 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:23:03.134785 8160 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:37035
I20260620 20:23:03.134806 8294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:37035 every 8 connection(s)
I20260620 20:23:03.135949 8160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:23:03.138110 8295 heartbeater.cc:347] Connected to a master server at 127.31.235.190:45937
I20260620 20:23:03.138353 8295 heartbeater.cc:464] Registering TS with master...
I20260620 20:23:03.138967 8295 heartbeater.cc:511] Master 127.31.235.190:45937 requested a full tablet report, sending...
I20260620 20:23:03.140290 7809 ts_manager.cc:194] Registered new tserver with Master: df43785430814849be835b580aabb724 (127.31.235.131:37035)
I20260620 20:23:03.141273 7809 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:49859
I20260620 20:23:03.145865 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 8160
I20260620 20:23:03.146037 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:23:03.146474 32686 external_mini_cluster.cc:1428] Setting key 423c9bc3414dc278cab32c9f6d6b58b9
I20260620 20:23:03.149081 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 20:23:03 dist-test-slave-zzwf krb5kdc[7765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986983, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 20:23:03 dist-test-slave-zzwf krb5kdc[7765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:03 dist-test-slave-zzwf krb5kdc[7765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:23:03.199133 7905 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:03.187806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58060 (local address 127.31.235.190:45937)
0620 20:23:03.188028 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:03.188035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:03.188231 (+ 196us) server_negotiation.cc:407] Connection header received
0620 20:23:03.188381 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:03.188385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:03.188449 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:03.188534 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:03.189853 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.190567 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.191810 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.192267 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.192384 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.198897 (+ 6513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58060: BlockingRecv error: recv got EOF from 127.0.0.1:58060 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 20 20:23:03 dist-test-slave-zzwf krb5kdc[7765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986983, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM
I20260620 20:23:03.216239 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.202344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58064 (local address 127.31.235.190:45937)
0620 20:23:03.202588 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:03.202596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:03.202894 (+ 298us) server_negotiation.cc:407] Connection header received
0620 20:23:03.203198 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:03.203202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:03.203276 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:03.203365 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:03.204650 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.205384 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.206642 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.207113 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.207217 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.210917 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:03.210922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:03.210926 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:03.210960 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:03.213723 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:03.214443 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:03.214447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:03.214449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:03.214558 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:03.214925 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:03.214929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:03.214932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:03.215221 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:03.215444 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:03.215646 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 20:23:03.215809 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:23:03.220192 7809 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:23:03.227479 7809 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:23:03.250684 8313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.239151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:39743 (local address 127.0.0.1:53518)
0620 20:23:03.240196 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:03.240247 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:03.240441 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:03.241744 (+ 1303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:03.241751 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:03.241779 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:23:03.242135 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:03.242156 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.244579 (+ 2423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.244588 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:03.247233 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:03.247252 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.249678 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.249684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:03.249896 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.249955 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 20:23:03.250077 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 20:23:03.250277 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":2299,"threads_started":1}
I20260620 20:23:03.252746 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.239904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53518 (local address 127.31.235.130:39743)
0620 20:23:03.241043 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:03.241055 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:03.241087 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:23:03.241207 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:03.241218 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:03.241526 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:03.242358 (+ 832us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:23:03.242560 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.244413 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.247473 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.249435 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.251816 (+ 2381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.252059 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:03.252136 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 20:23:03.252372 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":947,"thread_start_us":823,"threads_started":1}
I20260620 20:23:03.257331 8086 tablet_service.cc:1467] Processing CreateTablet for tablet 63413860a33d4a06b568f06291e41104 (DEFAULT_TABLE table=test-table [id=bdeda875448f49a29a2f77c9de73c3e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:23:03.259521 8086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63413860a33d4a06b568f06291e41104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:03.266942 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.239558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:35757 (local address 127.0.0.1:60266)
0620 20:23:03.258618 (+ 19060us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:03.258646 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:03.258807 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:03.259507 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:03.259514 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:03.259543 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:23:03.259882 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:03.259894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.262026 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.262032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:03.264406 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:03.264425 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.265924 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.265930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:03.266121 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.266172 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 20:23:03.266287 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 20:23:03.266441 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18420,"mutex_wait_us":27,"thread_start_us":18219,"threads_started":1}
I20260620 20:23:03.267213 8314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.239423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57180 (local address 127.31.235.131:37035)
0620 20:23:03.259434 (+ 20011us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:03.259446 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:03.259480 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:23:03.259681 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:03.259697 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:03.259935 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:03.260623 (+ 688us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:23:03.260929 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.262468 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.264698 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.266155 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.266342 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.266488 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:03.266736 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 20:23:03.266902 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19817,"thread_start_us":20252,"threads_started":1}
I20260620 20:23:03.267627 8313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.239161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:37035 (local address 127.0.0.1:57180)
0620 20:23:03.251443 (+ 12282us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:03.251466 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:03.251577 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:03.260182 (+ 8605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:03.260189 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:03.260220 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:23:03.260440 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:03.260453 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.262645 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.262651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:03.264509 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:03.264529 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.266315 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.266321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:03.266515 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.266556 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 20:23:03.266659 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 20:23:03.266808 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12148,"mutex_wait_us":3,"thread_start_us":19092,"threads_started":1}
I20260620 20:23:03.267797 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.239805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60266 (local address 127.31.235.129:35757)
0620 20:23:03.255744 (+ 15939us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:03.255751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:03.258754 (+ 3003us) server_negotiation.cc:407] Connection header received
0620 20:23:03.258976 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:03.258988 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:03.259273 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:03.259498 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:23:03.260064 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.261884 (+ 1820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.264579 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.265707 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.265916 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.266117 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:03.267266 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0620 20:23:03.267450 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15778,"thread_start_us":15639,"threads_started":1}
I20260620 20:23:03.271519 8225 tablet_service.cc:1467] Processing CreateTablet for tablet 63413860a33d4a06b568f06291e41104 (DEFAULT_TABLE table=test-table [id=bdeda875448f49a29a2f77c9de73c3e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:23:03.271569 7947 tablet_service.cc:1467] Processing CreateTablet for tablet 63413860a33d4a06b568f06291e41104 (DEFAULT_TABLE table=test-table [id=bdeda875448f49a29a2f77c9de73c3e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:23:03.272923 8225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63413860a33d4a06b568f06291e41104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:03.272938 7947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63413860a33d4a06b568f06291e41104. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:03.274578 8319 tablet_bootstrap.cc:492] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Bootstrap starting.
I20260620 20:23:03.278366 8319 tablet_bootstrap.cc:654] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:03.279704 8319 log.cc:824] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:23:03.283034 8319 tablet_bootstrap.cc:492] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f: No bootstrap required, opened a new log
I20260620 20:23:03.283334 8319 ts_tablet_manager.cc:1397] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 20:23:03.284950 8321 tablet_bootstrap.cc:492] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e: Bootstrap starting.
I20260620 20:23:03.288300 8322 tablet_bootstrap.cc:492] T 63413860a33d4a06b568f06291e41104 P df43785430814849be835b580aabb724: Bootstrap starting.
I20260620 20:23:03.290162 8321 tablet_bootstrap.cc:654] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:03.291007 8322 tablet_bootstrap.cc:654] T 63413860a33d4a06b568f06291e41104 P df43785430814849be835b580aabb724: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:03.291517 8321 log.cc:824] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:23:03.291989 8322 log.cc:824] T 63413860a33d4a06b568f06291e41104 P df43785430814849be835b580aabb724: Log is configured to *not* fsync() on all Append() calls
I20260620 20:23:03.294796 8321 tablet_bootstrap.cc:492] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e: No bootstrap required, opened a new log
I20260620 20:23:03.295002 8322 tablet_bootstrap.cc:492] T 63413860a33d4a06b568f06291e41104 P df43785430814849be835b580aabb724: No bootstrap required, opened a new log
I20260620 20:23:03.295086 8321 ts_tablet_manager.cc:1397] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 20:23:03.295388 8322 ts_tablet_manager.cc:1397] T 63413860a33d4a06b568f06291e41104 P df43785430814849be835b580aabb724: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.005s
I20260620 20:23:03.295588 8319 raft_consensus.cc:348] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } }
I20260620 20:23:03.296061 8319 raft_consensus.cc:374] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:03.296185 8319 raft_consensus.cc:729] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e64797c14e4e8e8f4bc6eb2463ae1f, State: Initialized, Role: FOLLOWER
I20260620 20:23:03.297001 8319 consensus_queue.cc:260] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } }
I20260620 20:23:03.298269 8156 heartbeater.cc:503] Master 127.31.235.190:45937 was elected leader, sending a full tablet report...
I20260620 20:23:03.298929 8319 ts_tablet_manager.cc:1428] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260620 20:23:03.302335 8325 raft_consensus.cc:482] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:23:03.302668 8325 raft_consensus.cc:504] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } }
I20260620 20:23:03.305716 8321 raft_consensus.cc:348] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } }
I20260620 20:23:03.306710 8321 raft_consensus.cc:374] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:03.306805 8321 raft_consensus.cc:729] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db89954f95d646f48ec84241c5307e2e, State: Initialized, Role: FOLLOWER
I20260620 20:23:03.307564 8321 consensus_queue.cc:260] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } }
I20260620 20:23:03.308866 8322 raft_consensus.cc:348] T 63413860a33d4a06b568f06291e41104 P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } }
I20260620 20:23:03.309473 8322 raft_consensus.cc:374] T 63413860a33d4a06b568f06291e41104 P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:03.309624 8322 raft_consensus.cc:729] T 63413860a33d4a06b568f06291e41104 P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df43785430814849be835b580aabb724, State: Initialized, Role: FOLLOWER
I20260620 20:23:03.310529 8322 consensus_queue.cc:260] T 63413860a33d4a06b568f06291e41104 P df43785430814849be835b580aabb724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } }
I20260620 20:23:03.311764 8325 leader_election.cc:288] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df43785430814849be835b580aabb724 (127.31.235.131:37035), db89954f95d646f48ec84241c5307e2e (127.31.235.129:35757)
I20260620 20:23:03.314364 8295 heartbeater.cc:503] Master 127.31.235.190:45937 was elected leader, sending a full tablet report...
I20260620 20:23:03.315794 8322 ts_tablet_manager.cc:1428] T 63413860a33d4a06b568f06291e41104 P df43785430814849be835b580aabb724: Time spent starting tablet: real 0.020s user 0.011s sys 0.002s
I20260620 20:23:03.319156 8321 ts_tablet_manager.cc:1428] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e: Time spent starting tablet: real 0.024s user 0.013s sys 0.010s
I20260620 20:23:03.323168 8314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.306133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:55845 (local address 127.31.235.131:37035)
0620 20:23:03.306334 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:03.306343 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:03.311932 (+ 5589us) server_negotiation.cc:407] Connection header received
0620 20:23:03.312842 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:03.312852 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:03.312959 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:03.313062 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:23:03.313872 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.315256 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.320772 (+ 5516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.322139 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.322322 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.322409 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:03.322781 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 20:23:03.322912 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 20:23:03.323721 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.307560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:34937 (local address 127.31.235.129:35757)
0620 20:23:03.307786 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:03.307799 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:03.313447 (+ 5648us) server_negotiation.cc:407] Connection header received
0620 20:23:03.316447 (+ 3000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:03.316457 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:03.316564 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:03.316638 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:23:03.317712 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.318919 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.321640 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.322640 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.322803 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.322867 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:03.323376 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 20:23:03.323497 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:23:03.323872 8327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.305331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:37035 (local address 127.31.235.130:55845)
0620 20:23:03.311782 (+ 6451us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:03.311817 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:03.312623 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:03.313294 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:03.313297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:03.313320 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:23:03.313711 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:03.313721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.315810 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.315817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:03.320612 (+ 4795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:03.320624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.322387 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.322394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:03.322581 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.322619 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 20:23:03.323350 (+ 731us) client_negotiation.cc:240] Negotiation successful
0620 20:23:03.323491 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6259,"mutex_wait_us":56,"thread_start_us":7595,"threads_started":1}
I20260620 20:23:03.324295 8326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.307634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:35757 (local address 127.31.235.130:34937)
0620 20:23:03.313302 (+ 5668us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:03.313326 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:03.316161 (+ 2835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:03.316867 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:03.316875 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:03.316913 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:23:03.317468 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:03.317484 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.319111 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.319117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:03.321433 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:03.321449 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.323000 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.323006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:03.323166 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.323222 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 20:23:03.323898 (+ 676us) client_negotiation.cc:240] Negotiation successful
0620 20:23:03.324043 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5492,"mutex_wait_us":1,"thread_start_us":3695,"threads_started":1}
I20260620 20:23:03.324862 8250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63413860a33d4a06b568f06291e41104" candidate_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df43785430814849be835b580aabb724" is_pre_election: true
I20260620 20:23:03.325168 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63413860a33d4a06b568f06291e41104" candidate_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db89954f95d646f48ec84241c5307e2e" is_pre_election: true
I20260620 20:23:03.325414 8250 raft_consensus.cc:2454] T 63413860a33d4a06b568f06291e41104 P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0e64797c14e4e8e8f4bc6eb2463ae1f in term 0.
I20260620 20:23:03.325676 7970 raft_consensus.cc:2454] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0e64797c14e4e8e8f4bc6eb2463ae1f in term 0.
I20260620 20:23:03.326153 8037 leader_election.cc:302] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e64797c14e4e8e8f4bc6eb2463ae1f, df43785430814849be835b580aabb724; no voters:
I20260620 20:23:03.326700 8325 raft_consensus.cc:2781] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:23:03.326872 8325 raft_consensus.cc:482] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:23:03.326952 8325 raft_consensus.cc:3037] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:03.329296 8325 raft_consensus.cc:504] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } }
I20260620 20:23:03.329948 8325 leader_election.cc:288] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f [CANDIDATE]: Term 1 election: Requested vote from peers df43785430814849be835b580aabb724 (127.31.235.131:37035), db89954f95d646f48ec84241c5307e2e (127.31.235.129:35757)
I20260620 20:23:03.330360 8250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63413860a33d4a06b568f06291e41104" candidate_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df43785430814849be835b580aabb724"
I20260620 20:23:03.330427 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63413860a33d4a06b568f06291e41104" candidate_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db89954f95d646f48ec84241c5307e2e"
I20260620 20:23:03.330547 8250 raft_consensus.cc:3037] T 63413860a33d4a06b568f06291e41104 P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:03.330652 7970 raft_consensus.cc:3037] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:03.332751 7970 raft_consensus.cc:2454] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e64797c14e4e8e8f4bc6eb2463ae1f in term 1.
I20260620 20:23:03.332973 8250 raft_consensus.cc:2454] T 63413860a33d4a06b568f06291e41104 P df43785430814849be835b580aabb724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0e64797c14e4e8e8f4bc6eb2463ae1f in term 1.
I20260620 20:23:03.333201 8038 leader_election.cc:302] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e64797c14e4e8e8f4bc6eb2463ae1f, db89954f95d646f48ec84241c5307e2e; no voters:
I20260620 20:23:03.333525 8325 raft_consensus.cc:2781] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:23:03.333943 8325 raft_consensus.cc:686] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 1 LEADER]: Becoming Leader. State: Replica: d0e64797c14e4e8e8f4bc6eb2463ae1f, State: Running, Role: LEADER
I20260620 20:23:03.334352 8325 consensus_queue.cc:237] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } }
I20260620 20:23:03.339159 7809 catalog_manager.cc:5495] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f reported cstate change: term changed from 0 to 1, leader changed from <none> to d0e64797c14e4e8e8f4bc6eb2463ae1f (127.31.235.130). New cstate: current_term: 1 leader_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } health_report { overall_health: UNKNOWN } } }
W20260620 20:23:03.354992 8018 tablet.cc:2369] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:23:03.357625 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.352237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53530 (local address 127.31.235.130:39743)
0620 20:23:03.352388 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:03.352392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:03.352409 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:23:03.352652 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:03.352656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:03.352726 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:03.352849 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:23:03.353193 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.353985 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.355530 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.355920 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.356010 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.356367 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:23:03.356946 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:23:03.357054 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:03.357219 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 20:23:03.357283 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 20:23:03.365048 8250 raft_consensus.cc:1261] T 63413860a33d4a06b568f06291e41104 P df43785430814849be835b580aabb724 [term 1 FOLLOWER]: Refusing update from remote peer d0e64797c14e4e8e8f4bc6eb2463ae1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:23:03.365072 7970 raft_consensus.cc:1261] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e [term 1 FOLLOWER]: Refusing update from remote peer d0e64797c14e4e8e8f4bc6eb2463ae1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:23:03.366019 8330 consensus_queue.cc:1035] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:23:03.366390 8325 consensus_queue.cc:1035] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 20:23:03.373924 8157 tablet.cc:2369] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:23:03.390372 7810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:23:03.390590 7810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:23:03.394052 7810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5: Sending DeleteTablet for 3 replicas of tablet 63413860a33d4a06b568f06291e41104
I20260620 20:23:03.395215 8225 tablet_service.cc:1514] Processing DeleteTablet for tablet 63413860a33d4a06b568f06291e41104 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:23:03 UTC) from {username='oryx', principal='oryx/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:57180
I20260620 20:23:03.395291 8086 tablet_service.cc:1514] Processing DeleteTablet for tablet 63413860a33d4a06b568f06291e41104 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:23:03 UTC) from {username='oryx', principal='oryx/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:53518
I20260620 20:23:03.395431 7947 tablet_service.cc:1514] Processing DeleteTablet for tablet 63413860a33d4a06b568f06291e41104 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:23:03 UTC) from {username='oryx', principal='oryx/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:60266
I20260620 20:23:03.396595 7809 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:23:03.397408 7809 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:23:03.407516 7946 tablet_service.cc:1467] Processing CreateTablet for tablet 6e6350f9a87b44f682d9af68a83a32b9 (DEFAULT_TABLE table=test-table [id=cac086efffaa482894b34057c97bd776]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:23:03.407936 8224 tablet_service.cc:1467] Processing CreateTablet for tablet 6e6350f9a87b44f682d9af68a83a32b9 (DEFAULT_TABLE table=test-table [id=cac086efffaa482894b34057c97bd776]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:23:03.407938 8085 tablet_service.cc:1467] Processing CreateTablet for tablet 6e6350f9a87b44f682d9af68a83a32b9 (DEFAULT_TABLE table=test-table [id=cac086efffaa482894b34057c97bd776]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:23:03.408071 7946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e6350f9a87b44f682d9af68a83a32b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:03.408442 8224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e6350f9a87b44f682d9af68a83a32b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:03.408474 8085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e6350f9a87b44f682d9af68a83a32b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:03.410014 8344 tablet_replica.cc:330] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e: stopping tablet replica
I20260620 20:23:03.410470 8344 raft_consensus.cc:2229] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:23:03.410991 8344 raft_consensus.cc:2258] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:23:03.413033 8344 ts_tablet_manager.cc:1905] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:23:03.415038 8321 tablet_bootstrap.cc:492] T 6e6350f9a87b44f682d9af68a83a32b9 P db89954f95d646f48ec84241c5307e2e: Bootstrap starting.
I20260620 20:23:03.417125 8319 tablet_bootstrap.cc:492] T 6e6350f9a87b44f682d9af68a83a32b9 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Bootstrap starting.
I20260620 20:23:03.417892 8321 tablet_bootstrap.cc:654] T 6e6350f9a87b44f682d9af68a83a32b9 P db89954f95d646f48ec84241c5307e2e: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:03.418763 8319 tablet_bootstrap.cc:654] T 6e6350f9a87b44f682d9af68a83a32b9 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:03.420480 8344 ts_tablet_manager.cc:1918] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:23:03.420660 8344 log.cc:1196] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/63413860a33d4a06b568f06291e41104
I20260620 20:23:03.420958 8322 tablet_bootstrap.cc:492] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724: Bootstrap starting.
I20260620 20:23:03.421254 8344 ts_tablet_manager.cc:1939] T 63413860a33d4a06b568f06291e41104 P db89954f95d646f48ec84241c5307e2e: Deleting consensus metadata
I20260620 20:23:03.422844 8322 tablet_bootstrap.cc:654] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:03.423283 7795 catalog_manager.cc:4841] TS db89954f95d646f48ec84241c5307e2e (127.31.235.129:35757): tablet 63413860a33d4a06b568f06291e41104 (table test-table [id=bdeda875448f49a29a2f77c9de73c3e2]) successfully deleted
I20260620 20:23:03.425057 8319 tablet_bootstrap.cc:492] T 6e6350f9a87b44f682d9af68a83a32b9 P d0e64797c14e4e8e8f4bc6eb2463ae1f: No bootstrap required, opened a new log
I20260620 20:23:03.425204 8319 ts_tablet_manager.cc:1397] T 6e6350f9a87b44f682d9af68a83a32b9 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 20:23:03.426095 8319 raft_consensus.cc:348] T 6e6350f9a87b44f682d9af68a83a32b9 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:03.426344 8319 raft_consensus.cc:374] T 6e6350f9a87b44f682d9af68a83a32b9 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:03.426452 8319 raft_consensus.cc:729] T 6e6350f9a87b44f682d9af68a83a32b9 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e64797c14e4e8e8f4bc6eb2463ae1f, State: Initialized, Role: FOLLOWER
I20260620 20:23:03.426748 8319 consensus_queue.cc:260] T 6e6350f9a87b44f682d9af68a83a32b9 P d0e64797c14e4e8e8f4bc6eb2463ae1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:03.430372 8319 ts_tablet_manager.cc:1428] T 6e6350f9a87b44f682d9af68a83a32b9 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260620 20:23:03.431664 8343 tablet_replica.cc:330] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f: stopping tablet replica
I20260620 20:23:03.432245 8343 raft_consensus.cc:2229] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:23:03.432950 8343 raft_consensus.cc:2258] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:23:03.433851 8322 tablet_bootstrap.cc:492] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724: No bootstrap required, opened a new log
I20260620 20:23:03.434041 8322 ts_tablet_manager.cc:1397] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260620 20:23:03.434871 8322 raft_consensus.cc:348] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:03.435070 8343 ts_tablet_manager.cc:1905] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:23:03.435181 8322 raft_consensus.cc:374] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:03.435319 8322 raft_consensus.cc:729] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df43785430814849be835b580aabb724, State: Initialized, Role: FOLLOWER
I20260620 20:23:03.435719 8322 consensus_queue.cc:260] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:03.438439 8322 ts_tablet_manager.cc:1428] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 20:23:03.441643 8321 tablet_bootstrap.cc:492] T 6e6350f9a87b44f682d9af68a83a32b9 P db89954f95d646f48ec84241c5307e2e: No bootstrap required, opened a new log
I20260620 20:23:03.441839 8321 ts_tablet_manager.cc:1397] T 6e6350f9a87b44f682d9af68a83a32b9 P db89954f95d646f48ec84241c5307e2e: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.001s
I20260620 20:23:03.442183 8343 ts_tablet_manager.cc:1918] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:23:03.442284 8342 tablet_replica.cc:330] T 63413860a33d4a06b568f06291e41104 P df43785430814849be835b580aabb724: stopping tablet replica
I20260620 20:23:03.442355 8343 log.cc:1196] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/63413860a33d4a06b568f06291e41104
I20260620 20:23:03.442694 8342 raft_consensus.cc:2229] T 63413860a33d4a06b568f06291e41104 P df43785430814849be835b580aabb724 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:23:03.442955 8343 ts_tablet_manager.cc:1939] T 63413860a33d4a06b568f06291e41104 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Deleting consensus metadata
I20260620 20:23:03.443157 8342 raft_consensus.cc:2258] T 63413860a33d4a06b568f06291e41104 P df43785430814849be835b580aabb724 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:23:03.443996 8321 raft_consensus.cc:348] T 6e6350f9a87b44f682d9af68a83a32b9 P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:03.444279 8321 raft_consensus.cc:374] T 6e6350f9a87b44f682d9af68a83a32b9 P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:03.444416 8321 raft_consensus.cc:729] T 6e6350f9a87b44f682d9af68a83a32b9 P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db89954f95d646f48ec84241c5307e2e, State: Initialized, Role: FOLLOWER
I20260620 20:23:03.444803 8321 consensus_queue.cc:260] T 6e6350f9a87b44f682d9af68a83a32b9 P db89954f95d646f48ec84241c5307e2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:03.445003 8342 ts_tablet_manager.cc:1905] T 63413860a33d4a06b568f06291e41104 P df43785430814849be835b580aabb724: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:23:03.445274 7797 catalog_manager.cc:4841] TS d0e64797c14e4e8e8f4bc6eb2463ae1f (127.31.235.130:39743): tablet 63413860a33d4a06b568f06291e41104 (table test-table [id=bdeda875448f49a29a2f77c9de73c3e2]) successfully deleted
I20260620 20:23:03.445551 8321 ts_tablet_manager.cc:1428] T 6e6350f9a87b44f682d9af68a83a32b9 P db89954f95d646f48ec84241c5307e2e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 20:23:03.451795 8342 ts_tablet_manager.cc:1918] T 63413860a33d4a06b568f06291e41104 P df43785430814849be835b580aabb724: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:23:03.451954 8342 log.cc:1196] T 63413860a33d4a06b568f06291e41104 P df43785430814849be835b580aabb724: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/63413860a33d4a06b568f06291e41104
I20260620 20:23:03.452539 8342 ts_tablet_manager.cc:1939] T 63413860a33d4a06b568f06291e41104 P df43785430814849be835b580aabb724: Deleting consensus metadata
I20260620 20:23:03.454236 7794 catalog_manager.cc:4841] TS df43785430814849be835b580aabb724 (127.31.235.131:37035): tablet 63413860a33d4a06b568f06291e41104 (table test-table [id=bdeda875448f49a29a2f77c9de73c3e2]) successfully deleted
I20260620 20:23:03.621431 8329 raft_consensus.cc:482] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:23:03.621693 8329 raft_consensus.cc:504] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:03.623080 8329 leader_election.cc:288] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db89954f95d646f48ec84241c5307e2e (127.31.235.129:35757), d0e64797c14e4e8e8f4bc6eb2463ae1f (127.31.235.130:39743)
W20260620 20:23:03.638500 8296 tablet.cc:2369] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:23:03.641819 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.623768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:47727 (local address 127.31.235.129:35757)
0620 20:23:03.623953 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:03.623960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:03.634171 (+ 10211us) server_negotiation.cc:407] Connection header received
0620 20:23:03.634426 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:03.634432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:03.634521 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:03.634638 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:23:03.635346 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.636437 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.639371 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.640621 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.640775 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.640852 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:03.641378 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 20:23:03.641528 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:23:03.642063 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.623794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:43947 (local address 127.31.235.130:39743)
0620 20:23:03.624021 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:03.624030 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:03.635031 (+ 11001us) server_negotiation.cc:407] Connection header received
0620 20:23:03.635291 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:03.635298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:03.635371 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:03.635440 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:23:03.636084 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.637239 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.639616 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.640907 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.641016 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.641086 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:03.641652 (+ 566us) server_negotiation.cc:299] Negotiation successful
0620 20:23:03.641777 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 20:23:03.642568 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.623371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:35757 (local address 127.31.235.131:47727)
0620 20:23:03.634020 (+ 10649us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:03.634049 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:03.634278 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:03.634945 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:03.634951 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:03.634982 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:23:03.635193 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:03.635205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.636624 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.636629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:03.639198 (+ 2569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:03.639210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.640989 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.640994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:03.641183 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.641227 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 20:23:03.641985 (+ 758us) client_negotiation.cc:240] Negotiation successful
0620 20:23:03.642203 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10466,"thread_start_us":11026,"threads_started":1}
I20260620 20:23:03.643122 8349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.623462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:39743 (local address 127.31.235.131:43947)
0620 20:23:03.634942 (+ 11480us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:03.634957 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:03.635118 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:03.635617 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:03.635621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:03.635644 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:23:03.635922 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:03.635932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.637412 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.637417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:03.639463 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:03.639474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.641281 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.641290 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:03.641462 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.641521 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 20:23:03.642470 (+ 949us) client_negotiation.cc:240] Negotiation successful
0620 20:23:03.642802 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11383,"thread_start_us":11262,"threads_started":1}
I20260620 20:23:03.643231 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6350f9a87b44f682d9af68a83a32b9" candidate_uuid: "df43785430814849be835b580aabb724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db89954f95d646f48ec84241c5307e2e" is_pre_election: true
I20260620 20:23:03.643481 7970 raft_consensus.cc:2454] T 6e6350f9a87b44f682d9af68a83a32b9 P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df43785430814849be835b580aabb724 in term 0.
I20260620 20:23:03.643961 8177 leader_election.cc:302] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db89954f95d646f48ec84241c5307e2e, df43785430814849be835b580aabb724; no voters:
I20260620 20:23:03.644142 8110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6350f9a87b44f682d9af68a83a32b9" candidate_uuid: "df43785430814849be835b580aabb724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" is_pre_election: true
I20260620 20:23:03.644397 8329 raft_consensus.cc:2781] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:23:03.644522 8329 raft_consensus.cc:482] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:23:03.644603 8329 raft_consensus.cc:3037] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:03.644590 8110 raft_consensus.cc:2454] T 6e6350f9a87b44f682d9af68a83a32b9 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df43785430814849be835b580aabb724 in term 0.
I20260620 20:23:03.647825 8329 raft_consensus.cc:504] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:03.648522 8329 leader_election.cc:288] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 [CANDIDATE]: Term 1 election: Requested vote from peers db89954f95d646f48ec84241c5307e2e (127.31.235.129:35757), d0e64797c14e4e8e8f4bc6eb2463ae1f (127.31.235.130:39743)
I20260620 20:23:03.648890 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6350f9a87b44f682d9af68a83a32b9" candidate_uuid: "df43785430814849be835b580aabb724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db89954f95d646f48ec84241c5307e2e"
I20260620 20:23:03.649004 8110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e6350f9a87b44f682d9af68a83a32b9" candidate_uuid: "df43785430814849be835b580aabb724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f"
I20260620 20:23:03.649120 7970 raft_consensus.cc:3037] T 6e6350f9a87b44f682d9af68a83a32b9 P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:03.649219 8110 raft_consensus.cc:3037] T 6e6350f9a87b44f682d9af68a83a32b9 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:03.651993 7970 raft_consensus.cc:2454] T 6e6350f9a87b44f682d9af68a83a32b9 P db89954f95d646f48ec84241c5307e2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df43785430814849be835b580aabb724 in term 1.
I20260620 20:23:03.652382 8177 leader_election.cc:302] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db89954f95d646f48ec84241c5307e2e, df43785430814849be835b580aabb724; no voters:
I20260620 20:23:03.652877 8110 raft_consensus.cc:2454] T 6e6350f9a87b44f682d9af68a83a32b9 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df43785430814849be835b580aabb724 in term 1.
I20260620 20:23:03.653641 8329 raft_consensus.cc:2781] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:23:03.654114 8329 raft_consensus.cc:686] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 [term 1 LEADER]: Becoming Leader. State: Replica: df43785430814849be835b580aabb724, State: Running, Role: LEADER
I20260620 20:23:03.654567 8329 consensus_queue.cc:237] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:03.658102 7810 catalog_manager.cc:5495] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 reported cstate change: term changed from 0 to 1, leader changed from <none> to df43785430814849be835b580aabb724 (127.31.235.131). New cstate: current_term: 1 leader_uuid: "df43785430814849be835b580aabb724" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } health_report { overall_health: HEALTHY } } }
Jun 20 20:23:03 dist-test-slave-zzwf krb5kdc[7765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986981, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.190@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM
I20260620 20:23:03.749294 8358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.732206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58076 (local address 127.31.235.190:45937)
0620 20:23:03.734396 (+ 2190us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:03.734401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:03.734421 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:23:03.734466 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:03.734470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:03.734521 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:03.734621 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:03.736123 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.736918 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.738813 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.739343 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.739444 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.743507 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:03.743512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:03.743515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:03.743558 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:03.746329 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:03.747223 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:03.747227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:03.747229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:03.747360 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:03.747747 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:03.747750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:03.747752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:03.748011 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:03.748142 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:03.748696 (+ 554us) server_negotiation.cc:299] Negotiation successful
0620 20:23:03.748975 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":2100,"thread_start_us":1914,"threads_started":1}
I20260620 20:23:03.749382 8357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:03.732117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:45937 (local address 127.0.0.1:58076)
0620 20:23:03.734067 (+ 1950us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:03.734090 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:03.734255 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:03.734674 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:03.734682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:03.735587 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:03.735938 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:03.735951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.737056 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.737062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:03.738669 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:03.738684 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:03.739542 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:03.739550 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:03.739718 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:03.740468 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:03.740496 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:03.743285 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:23:03.746478 (+ 3193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:03.746491 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:03.746498 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:03.747074 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:03.747480 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:03.747485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:03.747489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:03.747645 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:03.748163 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:23:03.748172 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:23:03.748385 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 20:23:03.748561 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 20:23:03.748856 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1752,"thread_start_us":1571,"threads_started":1}
I20260620 20:23:03.755048 7810 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:58076:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 20:23:03.756083 7810 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:23:03.763175 7947 tablet_service.cc:1467] Processing CreateTablet for tablet da9efb558cde4fb9915b343b691be53b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d8bae9da12d40af96fe52998aa0c4f9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:23:03.763604 8086 tablet_service.cc:1467] Processing CreateTablet for tablet da9efb558cde4fb9915b343b691be53b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d8bae9da12d40af96fe52998aa0c4f9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:23:03.763830 7947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9efb558cde4fb9915b343b691be53b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:03.764192 8086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9efb558cde4fb9915b343b691be53b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:03.766181 8225 tablet_service.cc:1467] Processing CreateTablet for tablet da9efb558cde4fb9915b343b691be53b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2d8bae9da12d40af96fe52998aa0c4f9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 20:23:03.766709 8225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9efb558cde4fb9915b343b691be53b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:03.771554 8319 tablet_bootstrap.cc:492] T da9efb558cde4fb9915b343b691be53b P d0e64797c14e4e8e8f4bc6eb2463ae1f: Bootstrap starting.
I20260620 20:23:03.773337 8319 tablet_bootstrap.cc:654] T da9efb558cde4fb9915b343b691be53b P d0e64797c14e4e8e8f4bc6eb2463ae1f: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:03.773818 8322 tablet_bootstrap.cc:492] T da9efb558cde4fb9915b343b691be53b P df43785430814849be835b580aabb724: Bootstrap starting.
I20260620 20:23:03.775861 8322 tablet_bootstrap.cc:654] T da9efb558cde4fb9915b343b691be53b P df43785430814849be835b580aabb724: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:03.778033 8321 tablet_bootstrap.cc:492] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e: Bootstrap starting.
I20260620 20:23:03.779733 8321 tablet_bootstrap.cc:654] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:03.785549 8322 tablet_bootstrap.cc:492] T da9efb558cde4fb9915b343b691be53b P df43785430814849be835b580aabb724: No bootstrap required, opened a new log
I20260620 20:23:03.785781 8322 ts_tablet_manager.cc:1397] T da9efb558cde4fb9915b343b691be53b P df43785430814849be835b580aabb724: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260620 20:23:03.786785 8322 raft_consensus.cc:348] T da9efb558cde4fb9915b343b691be53b P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } }
I20260620 20:23:03.787117 8322 raft_consensus.cc:374] T da9efb558cde4fb9915b343b691be53b P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:03.787216 8322 raft_consensus.cc:729] T da9efb558cde4fb9915b343b691be53b P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df43785430814849be835b580aabb724, State: Initialized, Role: FOLLOWER
I20260620 20:23:03.787550 8322 consensus_queue.cc:260] T da9efb558cde4fb9915b343b691be53b P df43785430814849be835b580aabb724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } }
I20260620 20:23:03.788090 8329 tablet_replica.cc:439] T da9efb558cde4fb9915b343b691be53b P df43785430814849be835b580aabb724: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } }
I20260620 20:23:03.788442 8329 tablet_replica.cc:442] T da9efb558cde4fb9915b343b691be53b P df43785430814849be835b580aabb724: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:23:03.791990 8322 ts_tablet_manager.cc:1428] T da9efb558cde4fb9915b343b691be53b P df43785430814849be835b580aabb724: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260620 20:23:03.792995 8321 tablet_bootstrap.cc:492] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e: No bootstrap required, opened a new log
I20260620 20:23:03.793234 8321 ts_tablet_manager.cc:1397] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.003s
I20260620 20:23:03.795094 8321 raft_consensus.cc:348] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } }
I20260620 20:23:03.795372 8321 raft_consensus.cc:374] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:03.795455 8321 raft_consensus.cc:729] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db89954f95d646f48ec84241c5307e2e, State: Initialized, Role: FOLLOWER
I20260620 20:23:03.795722 8321 consensus_queue.cc:260] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } }
I20260620 20:23:03.796432 8321 ts_tablet_manager.cc:1428] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 20:23:03.796283 8328 tablet_replica.cc:439] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } }
I20260620 20:23:03.796716 8328 tablet_replica.cc:442] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:23:03.796947 8319 tablet_bootstrap.cc:492] T da9efb558cde4fb9915b343b691be53b P d0e64797c14e4e8e8f4bc6eb2463ae1f: No bootstrap required, opened a new log
I20260620 20:23:03.797140 8319 ts_tablet_manager.cc:1397] T da9efb558cde4fb9915b343b691be53b P d0e64797c14e4e8e8f4bc6eb2463ae1f: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.015s
I20260620 20:23:03.798084 8319 raft_consensus.cc:348] T da9efb558cde4fb9915b343b691be53b P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } }
I20260620 20:23:03.798334 8319 raft_consensus.cc:374] T da9efb558cde4fb9915b343b691be53b P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:03.798439 8319 raft_consensus.cc:729] T da9efb558cde4fb9915b343b691be53b P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e64797c14e4e8e8f4bc6eb2463ae1f, State: Initialized, Role: FOLLOWER
I20260620 20:23:03.798717 8319 consensus_queue.cc:260] T da9efb558cde4fb9915b343b691be53b P d0e64797c14e4e8e8f4bc6eb2463ae1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } }
I20260620 20:23:03.799178 8325 tablet_replica.cc:439] T da9efb558cde4fb9915b343b691be53b P d0e64797c14e4e8e8f4bc6eb2463ae1f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } }
I20260620 20:23:03.799466 8325 tablet_replica.cc:442] T da9efb558cde4fb9915b343b691be53b P d0e64797c14e4e8e8f4bc6eb2463ae1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:23:03.799986 8319 ts_tablet_manager.cc:1428] T da9efb558cde4fb9915b343b691be53b P d0e64797c14e4e8e8f4bc6eb2463ae1f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 20:23:04.012614 8328 raft_consensus.cc:482] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:23:04.012806 8328 raft_consensus.cc:504] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } }
I20260620 20:23:04.014523 8328 leader_election.cc:288] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df43785430814849be835b580aabb724 (127.31.235.131:37035), d0e64797c14e4e8e8f4bc6eb2463ae1f (127.31.235.130:39743)
I20260620 20:23:04.020931 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:04.014895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:37495 (local address 127.31.235.130:39743)
0620 20:23:04.015120 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:04.015130 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:04.015468 (+ 338us) server_negotiation.cc:407] Connection header received
0620 20:23:04.015819 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:04.015824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:04.015906 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:04.015956 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:23:04.016638 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.017414 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.019134 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.019949 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.020081 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:04.020186 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:04.020511 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 20:23:04.020654 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 20:23:04.020949 8362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:04.014769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:39743 (local address 127.31.235.129:37495)
0620 20:23:04.015336 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:04.015362 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:04.015604 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:04.016167 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:04.016170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:04.016189 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:23:04.016481 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:04.016489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.017558 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.017563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:04.018972 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:04.018985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.020149 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.020155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:04.020355 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:04.020383 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 20:23:04.020457 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 20:23:04.020593 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":345,"threads_started":1}
I20260620 20:23:04.020982 8364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:04.014938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:48507 (local address 127.31.235.131:37035)
0620 20:23:04.015966 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:04.015972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:04.015996 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:23:04.016049 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:04.016053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:04.016106 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:04.016215 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:23:04.016861 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.017649 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.019296 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.020041 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.020193 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:04.020269 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:04.020585 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 20:23:04.020719 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":912,"thread_start_us":811,"threads_started":1}
I20260620 20:23:04.021041 8363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:04.014778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:37035 (local address 127.31.235.129:48507)
0620 20:23:04.015681 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:04.015703 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:04.015823 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:04.016228 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:04.016236 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:04.016289 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:23:04.016706 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:04.016717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.017775 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.017779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:04.019152 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:04.019165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.020284 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.020289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:04.020409 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:04.020431 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:23:04.020546 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 20:23:04.020669 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":61,"thread_start_us":633,"threads_started":1}
I20260620 20:23:04.021842 8250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9efb558cde4fb9915b343b691be53b" candidate_uuid: "db89954f95d646f48ec84241c5307e2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df43785430814849be835b580aabb724" is_pre_election: true
I20260620 20:23:04.021919 8110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9efb558cde4fb9915b343b691be53b" candidate_uuid: "db89954f95d646f48ec84241c5307e2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" is_pre_election: true
I20260620 20:23:04.022105 8250 raft_consensus.cc:2454] T da9efb558cde4fb9915b343b691be53b P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db89954f95d646f48ec84241c5307e2e in term 0.
I20260620 20:23:04.022157 8110 raft_consensus.cc:2454] T da9efb558cde4fb9915b343b691be53b P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db89954f95d646f48ec84241c5307e2e in term 0.
I20260620 20:23:04.022560 7897 leader_election.cc:302] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db89954f95d646f48ec84241c5307e2e, df43785430814849be835b580aabb724; no voters:
I20260620 20:23:04.022933 8328 raft_consensus.cc:2781] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:23:04.023056 8328 raft_consensus.cc:482] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:23:04.023149 8328 raft_consensus.cc:3037] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:04.025166 8328 raft_consensus.cc:504] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } }
I20260620 20:23:04.025825 8328 leader_election.cc:288] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e [CANDIDATE]: Term 1 election: Requested vote from peers df43785430814849be835b580aabb724 (127.31.235.131:37035), d0e64797c14e4e8e8f4bc6eb2463ae1f (127.31.235.130:39743)
I20260620 20:23:04.026218 8250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9efb558cde4fb9915b343b691be53b" candidate_uuid: "db89954f95d646f48ec84241c5307e2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df43785430814849be835b580aabb724"
I20260620 20:23:04.026304 8110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9efb558cde4fb9915b343b691be53b" candidate_uuid: "db89954f95d646f48ec84241c5307e2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f"
I20260620 20:23:04.026464 8250 raft_consensus.cc:3037] T da9efb558cde4fb9915b343b691be53b P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:04.026512 8110 raft_consensus.cc:3037] T da9efb558cde4fb9915b343b691be53b P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:04.028424 8110 raft_consensus.cc:2454] T da9efb558cde4fb9915b343b691be53b P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db89954f95d646f48ec84241c5307e2e in term 1.
I20260620 20:23:04.028757 8250 raft_consensus.cc:2454] T da9efb558cde4fb9915b343b691be53b P df43785430814849be835b580aabb724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db89954f95d646f48ec84241c5307e2e in term 1.
I20260620 20:23:04.028867 7900 leader_election.cc:302] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e64797c14e4e8e8f4bc6eb2463ae1f, db89954f95d646f48ec84241c5307e2e; no voters:
I20260620 20:23:04.029207 8328 raft_consensus.cc:2781] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:23:04.029655 8328 raft_consensus.cc:686] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e [term 1 LEADER]: Becoming Leader. State: Replica: db89954f95d646f48ec84241c5307e2e, State: Running, Role: LEADER
I20260620 20:23:04.030023 8328 consensus_queue.cc:237] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } }
I20260620 20:23:04.031881 8365 tablet_replica.cc:439] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e: TxnStatusTablet state changed. Reason: New leader db89954f95d646f48ec84241c5307e2e. Latest consensus state: current_term: 1 leader_uuid: "db89954f95d646f48ec84241c5307e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } }
I20260620 20:23:04.032254 8365 tablet_replica.cc:442] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:23:04.032889 8367 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 20:23:04.033099 8367 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 20:23:04.033588 7810 catalog_manager.cc:5495] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e reported cstate change: term changed from 0 to 1, leader changed from <none> to db89954f95d646f48ec84241c5307e2e (127.31.235.129). New cstate: current_term: 1 leader_uuid: "db89954f95d646f48ec84241c5307e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } health_report { overall_health: UNKNOWN } } }
I20260620 20:23:04.070065 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:04.063968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60282 (local address 127.31.235.129:35757)
0620 20:23:04.064155 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:04.064160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:04.064244 (+ 84us) server_negotiation.cc:407] Connection header received
0620 20:23:04.064516 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:04.064520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:04.064606 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:04.064715 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:23:04.065133 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.065929 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.067882 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.068321 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.068407 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:04.068936 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:23:04.069431 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:23:04.069595 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:04.069744 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 20:23:04.069811 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:23:04.070164 8357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:04.063887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:35757 (local address 127.0.0.1:60282)
0620 20:23:04.064151 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:04.064173 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:04.064323 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:04.064775 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:04.064780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:04.064797 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:23:04.064995 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:04.065007 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.066073 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.066079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:04.067706 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:04.067720 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.068469 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.068478 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:04.068675 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:04.068747 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:23:04.069565 (+ 818us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:23:04.069587 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:23:04.069681 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 20:23:04.069844 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 20:23:04.075199 8250 raft_consensus.cc:1261] T da9efb558cde4fb9915b343b691be53b P df43785430814849be835b580aabb724 [term 1 FOLLOWER]: Refusing update from remote peer db89954f95d646f48ec84241c5307e2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:23:04.075578 8110 raft_consensus.cc:1261] T da9efb558cde4fb9915b343b691be53b P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 1 FOLLOWER]: Refusing update from remote peer db89954f95d646f48ec84241c5307e2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:23:04.076035 8365 consensus_queue.cc:1035] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:23:04.076946 8365 consensus_queue.cc:1035] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:23:04.083443 8329 tablet_replica.cc:439] T da9efb558cde4fb9915b343b691be53b P df43785430814849be835b580aabb724: TxnStatusTablet state changed. Reason: New leader db89954f95d646f48ec84241c5307e2e. Latest consensus state: current_term: 1 leader_uuid: "db89954f95d646f48ec84241c5307e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } }
I20260620 20:23:04.084393 8329 tablet_replica.cc:442] T da9efb558cde4fb9915b343b691be53b P df43785430814849be835b580aabb724: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:23:04.086505 8325 tablet_replica.cc:439] T da9efb558cde4fb9915b343b691be53b P d0e64797c14e4e8e8f4bc6eb2463ae1f: TxnStatusTablet state changed. Reason: New leader db89954f95d646f48ec84241c5307e2e. Latest consensus state: current_term: 1 leader_uuid: "db89954f95d646f48ec84241c5307e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } }
I20260620 20:23:04.086772 8325 tablet_replica.cc:442] T da9efb558cde4fb9915b343b691be53b P d0e64797c14e4e8e8f4bc6eb2463ae1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:23:04.087113 8371 mvcc.cc:205] Tried to move back new op lower bound from 7299018686763233280 to 7299018686590349312. Current Snapshot: MvccSnapshot[applied={T|T < 7299018686763233280}]
I20260620 20:23:04.088496 8329 tablet_replica.cc:439] T da9efb558cde4fb9915b343b691be53b P df43785430814849be835b580aabb724: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db89954f95d646f48ec84241c5307e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } }
I20260620 20:23:04.088752 8329 tablet_replica.cc:442] T da9efb558cde4fb9915b343b691be53b P df43785430814849be835b580aabb724: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:23:04.092989 8325 tablet_replica.cc:439] T da9efb558cde4fb9915b343b691be53b P d0e64797c14e4e8e8f4bc6eb2463ae1f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "db89954f95d646f48ec84241c5307e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } }
I20260620 20:23:04.093225 8325 tablet_replica.cc:442] T da9efb558cde4fb9915b343b691be53b P d0e64797c14e4e8e8f4bc6eb2463ae1f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 20:23:04.093608 8370 mvcc.cc:205] Tried to move back new op lower bound from 7299018686763233280 to 7299018686590349312. Current Snapshot: MvccSnapshot[applied={T|T < 7299018686763233280}]
I20260620 20:23:04.095072 8328 tablet_replica.cc:439] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db89954f95d646f48ec84241c5307e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } }
I20260620 20:23:04.095366 8328 tablet_replica.cc:442] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:23:04.096005 8328 tablet_replica.cc:439] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "db89954f95d646f48ec84241c5307e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } }
I20260620 20:23:04.096311 8328 tablet_replica.cc:442] T da9efb558cde4fb9915b343b691be53b P db89954f95d646f48ec84241c5307e2e: This TxnStatusTablet replica's current role is: LEADER
I20260620 20:23:04.120373 8364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:04.101146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57192 (local address 127.31.235.131:37035)
0620 20:23:04.101325 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:04.101333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:04.113675 (+ 12342us) server_negotiation.cc:407] Connection header received
0620 20:23:04.113877 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:04.113883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:04.113995 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:04.114147 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:23:04.114641 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.115710 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.117806 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.118455 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.118578 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:04.119055 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:23:04.119736 (+ 681us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:23:04.119870 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:04.119992 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 20:23:04.120072 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:23:04.143736 8329 consensus_queue.cc:1035] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:23:04.145149 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:04.135840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:50513 (local address 127.31.235.129:35757)
0620 20:23:04.136020 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:04.136026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:04.138637 (+ 2611us) server_negotiation.cc:407] Connection header received
0620 20:23:04.138842 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:04.138848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:04.138930 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:04.139033 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:23:04.139679 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.140807 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.142977 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.143681 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.143771 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:04.144365 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:23:04.144485 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:23:04.144545 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:04.144776 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 20:23:04.144852 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:23:04.146162 8382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:04.135823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:35757 (local address 127.31.235.131:50513)
0620 20:23:04.138527 (+ 2704us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:04.138551 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:04.138697 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:04.139069 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:04.139117 (+ 48us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:04.139134 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:23:04.139500 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:04.139510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.140980 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.140985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:04.142826 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:04.142839 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.143918 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.143923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:04.144111 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:04.144177 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:23:04.144630 (+ 453us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:23:04.144650 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 20:23:04.145730 (+ 1080us) client_negotiation.cc:240] Negotiation successful
0620 20:23:04.145866 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2471,"thread_start_us":2324,"threads_started":1}
I20260620 20:23:04.158061 8110 raft_consensus.cc:1261] T 6e6350f9a87b44f682d9af68a83a32b9 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 1 FOLLOWER]: Refusing update from remote peer df43785430814849be835b580aabb724: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:23:04.169428 8385 consensus_queue.cc:1035] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:23:04.205662 8364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:04.198670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:49095 (local address 127.31.235.131:37035)
0620 20:23:04.198883 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:04.198891 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:04.199220 (+ 329us) server_negotiation.cc:407] Connection header received
0620 20:23:04.199535 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:04.199541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:04.199639 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:04.199734 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:23:04.200396 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.201554 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.203519 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.204231 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.204351 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:04.204745 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:23:04.204905 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:23:04.204992 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:04.205208 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 20:23:04.205290 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:23:04.205804 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:04.198355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:37035 (local address 127.31.235.129:49095)
0620 20:23:04.199111 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:04.199134 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:04.199346 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:04.199782 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:04.199786 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:04.199796 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 20:23:04.200230 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:04.200241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.201687 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.201693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:04.203367 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:04.203383 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.204397 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.204402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:04.204522 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:04.204584 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 20:23:04.205001 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 20:23:04.205015 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 20:23:04.205180 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 20:23:04.205314 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":657,"threads_started":1}
I20260620 20:23:04.241362 7809 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:23:04.241676 7809 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:23:04.245522 7809 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5: Sending DeleteTablet for 3 replicas of tablet 6e6350f9a87b44f682d9af68a83a32b9
I20260620 20:23:04.246475 7947 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e6350f9a87b44f682d9af68a83a32b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:23:04 UTC) from {username='oryx', principal='oryx/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:60266
I20260620 20:23:04.246465 8225 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e6350f9a87b44f682d9af68a83a32b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:23:04 UTC) from {username='oryx', principal='oryx/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:57180
I20260620 20:23:04.247301 8086 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e6350f9a87b44f682d9af68a83a32b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:23:04 UTC) from {username='oryx', principal='oryx/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:53518
I20260620 20:23:04.290495 8394 tablet_replica.cc:330] T 6e6350f9a87b44f682d9af68a83a32b9 P d0e64797c14e4e8e8f4bc6eb2463ae1f: stopping tablet replica
I20260620 20:23:04.290737 8394 raft_consensus.cc:2229] T 6e6350f9a87b44f682d9af68a83a32b9 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:23:04.291004 8394 raft_consensus.cc:2258] T 6e6350f9a87b44f682d9af68a83a32b9 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:23:04.291641 8392 tablet_replica.cc:330] T 6e6350f9a87b44f682d9af68a83a32b9 P db89954f95d646f48ec84241c5307e2e: stopping tablet replica
I20260620 20:23:04.291851 8392 raft_consensus.cc:2229] T 6e6350f9a87b44f682d9af68a83a32b9 P db89954f95d646f48ec84241c5307e2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:23:04.292140 8392 raft_consensus.cc:2258] T 6e6350f9a87b44f682d9af68a83a32b9 P db89954f95d646f48ec84241c5307e2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:23:04.293093 8394 ts_tablet_manager.cc:1905] T 6e6350f9a87b44f682d9af68a83a32b9 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:23:04.295302 8393 tablet_replica.cc:330] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724: stopping tablet replica
I20260620 20:23:04.295547 8393 raft_consensus.cc:2229] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:23:04.296110 8393 raft_consensus.cc:2258] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:23:04.297214 8393 ts_tablet_manager.cc:1905] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:23:04.298671 8392 ts_tablet_manager.cc:1905] T 6e6350f9a87b44f682d9af68a83a32b9 P db89954f95d646f48ec84241c5307e2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:23:04.305100 8393 ts_tablet_manager.cc:1918] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:23:04.305334 8393 log.cc:1196] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/6e6350f9a87b44f682d9af68a83a32b9
I20260620 20:23:04.305954 8393 ts_tablet_manager.cc:1939] T 6e6350f9a87b44f682d9af68a83a32b9 P df43785430814849be835b580aabb724: Deleting consensus metadata
I20260620 20:23:04.307305 7794 catalog_manager.cc:4841] TS df43785430814849be835b580aabb724 (127.31.235.131:37035): tablet 6e6350f9a87b44f682d9af68a83a32b9 (table test-table [id=cac086efffaa482894b34057c97bd776]) successfully deleted
I20260620 20:23:04.307777 8394 ts_tablet_manager.cc:1918] T 6e6350f9a87b44f682d9af68a83a32b9 P d0e64797c14e4e8e8f4bc6eb2463ae1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:23:04.307915 8394 log.cc:1196] T 6e6350f9a87b44f682d9af68a83a32b9 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/6e6350f9a87b44f682d9af68a83a32b9
I20260620 20:23:04.308346 8394 ts_tablet_manager.cc:1939] T 6e6350f9a87b44f682d9af68a83a32b9 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Deleting consensus metadata
I20260620 20:23:04.309630 7797 catalog_manager.cc:4841] TS d0e64797c14e4e8e8f4bc6eb2463ae1f (127.31.235.130:39743): tablet 6e6350f9a87b44f682d9af68a83a32b9 (table test-table [id=cac086efffaa482894b34057c97bd776]) successfully deleted
I20260620 20:23:04.316824 8392 ts_tablet_manager.cc:1918] T 6e6350f9a87b44f682d9af68a83a32b9 P db89954f95d646f48ec84241c5307e2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:23:04.316980 8392 log.cc:1196] T 6e6350f9a87b44f682d9af68a83a32b9 P db89954f95d646f48ec84241c5307e2e: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/6e6350f9a87b44f682d9af68a83a32b9
I20260620 20:23:04.317482 8392 ts_tablet_manager.cc:1939] T 6e6350f9a87b44f682d9af68a83a32b9 P db89954f95d646f48ec84241c5307e2e: Deleting consensus metadata
I20260620 20:23:04.318745 7795 catalog_manager.cc:4841] TS db89954f95d646f48ec84241c5307e2e (127.31.235.129:35757): tablet 6e6350f9a87b44f682d9af68a83a32b9 (table test-table [id=cac086efffaa482894b34057c97bd776]) successfully deleted
Jun 20 20:23:04 dist-test-slave-zzwf krb5kdc[7765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 20:23:04 dist-test-slave-zzwf krb5kdc[7765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:04 dist-test-slave-zzwf krb5kdc[7765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:23:04.389415 8402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:04.359009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58080 (local address 127.31.235.190:45937)
0620 20:23:04.371309 (+ 12300us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:04.371316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:04.372036 (+ 720us) server_negotiation.cc:407] Connection header received
0620 20:23:04.372279 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:04.372288 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:04.372384 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:04.372524 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:04.374533 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.375736 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.377887 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.378538 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.378669 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:04.388993 (+ 10324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58080: BlockingRecv error: recv got EOF from 127.0.0.1:58080 (error 108)
Metrics: {"server-negotiator.queue_time_us":12127,"thread_start_us":12255,"threads_started":1}
Jun 20 20:23:04 dist-test-slave-zzwf krb5kdc[7765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM
I20260620 20:23:04.417321 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:04.388892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58088 (local address 127.31.235.190:45937)
0620 20:23:04.390012 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:04.390018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:04.397736 (+ 7718us) server_negotiation.cc:407] Connection header received
0620 20:23:04.397953 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:04.397959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:04.398051 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:04.398167 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:04.399859 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.401018 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.403216 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.403949 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.404096 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:04.409670 (+ 5574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:04.409676 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:04.409681 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:04.409724 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:04.413409 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:04.414379 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:04.414384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:04.414388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:04.414536 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:04.414965 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:04.414973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:04.414978 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:04.415373 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:04.415638 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:04.416626 (+ 988us) server_negotiation.cc:299] Negotiation successful
0620 20:23:04.416893 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1002,"thread_start_us":7810,"threads_started":1}
I20260620 20:23:04.421316 7809 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:23:04.422106 7809 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:23:04.430114 7947 tablet_service.cc:1467] Processing CreateTablet for tablet b582b055682c420e8633861baeb36a6d (DEFAULT_TABLE table=test-table [id=c932b6648413456488d1c52355296c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:23:04.430676 7947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b582b055682c420e8633861baeb36a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:04.430719 8225 tablet_service.cc:1467] Processing CreateTablet for tablet b582b055682c420e8633861baeb36a6d (DEFAULT_TABLE table=test-table [id=c932b6648413456488d1c52355296c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:23:04.431106 8225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b582b055682c420e8633861baeb36a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:04.433981 8086 tablet_service.cc:1467] Processing CreateTablet for tablet b582b055682c420e8633861baeb36a6d (DEFAULT_TABLE table=test-table [id=c932b6648413456488d1c52355296c0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:23:04.434432 8086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b582b055682c420e8633861baeb36a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:04.445348 8409 tablet_bootstrap.cc:492] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e: Bootstrap starting.
I20260620 20:23:04.446878 8410 tablet_bootstrap.cc:492] T b582b055682c420e8633861baeb36a6d P df43785430814849be835b580aabb724: Bootstrap starting.
I20260620 20:23:04.447707 8409 tablet_bootstrap.cc:654] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:04.449020 8411 tablet_bootstrap.cc:492] T b582b055682c420e8633861baeb36a6d P d0e64797c14e4e8e8f4bc6eb2463ae1f: Bootstrap starting.
I20260620 20:23:04.451088 8410 tablet_bootstrap.cc:654] T b582b055682c420e8633861baeb36a6d P df43785430814849be835b580aabb724: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:04.451140 8411 tablet_bootstrap.cc:654] T b582b055682c420e8633861baeb36a6d P d0e64797c14e4e8e8f4bc6eb2463ae1f: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:04.457568 8409 tablet_bootstrap.cc:492] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e: No bootstrap required, opened a new log
I20260620 20:23:04.457724 8409 ts_tablet_manager.cc:1397] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260620 20:23:04.458545 8409 raft_consensus.cc:348] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:04.458791 8409 raft_consensus.cc:374] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:04.458878 8409 raft_consensus.cc:729] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db89954f95d646f48ec84241c5307e2e, State: Initialized, Role: FOLLOWER
I20260620 20:23:04.459117 8409 consensus_queue.cc:260] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:04.459667 8409 ts_tablet_manager.cc:1428] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 20:23:04.463018 8411 tablet_bootstrap.cc:492] T b582b055682c420e8633861baeb36a6d P d0e64797c14e4e8e8f4bc6eb2463ae1f: No bootstrap required, opened a new log
I20260620 20:23:04.463166 8411 ts_tablet_manager.cc:1397] T b582b055682c420e8633861baeb36a6d P d0e64797c14e4e8e8f4bc6eb2463ae1f: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20260620 20:23:04.463939 8411 raft_consensus.cc:348] T b582b055682c420e8633861baeb36a6d P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:04.464192 8411 raft_consensus.cc:374] T b582b055682c420e8633861baeb36a6d P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:04.464280 8411 raft_consensus.cc:729] T b582b055682c420e8633861baeb36a6d P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e64797c14e4e8e8f4bc6eb2463ae1f, State: Initialized, Role: FOLLOWER
I20260620 20:23:04.464546 8411 consensus_queue.cc:260] T b582b055682c420e8633861baeb36a6d P d0e64797c14e4e8e8f4bc6eb2463ae1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:04.467173 8411 ts_tablet_manager.cc:1428] T b582b055682c420e8633861baeb36a6d P d0e64797c14e4e8e8f4bc6eb2463ae1f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260620 20:23:04.476922 8410 tablet_bootstrap.cc:492] T b582b055682c420e8633861baeb36a6d P df43785430814849be835b580aabb724: No bootstrap required, opened a new log
I20260620 20:23:04.477100 8410 ts_tablet_manager.cc:1397] T b582b055682c420e8633861baeb36a6d P df43785430814849be835b580aabb724: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.006s
I20260620 20:23:04.477947 8410 raft_consensus.cc:348] T b582b055682c420e8633861baeb36a6d P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:04.478238 8410 raft_consensus.cc:374] T b582b055682c420e8633861baeb36a6d P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:04.478343 8410 raft_consensus.cc:729] T b582b055682c420e8633861baeb36a6d P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df43785430814849be835b580aabb724, State: Initialized, Role: FOLLOWER
I20260620 20:23:04.478631 8410 consensus_queue.cc:260] T b582b055682c420e8633861baeb36a6d P df43785430814849be835b580aabb724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:04.479207 8410 ts_tablet_manager.cc:1428] T b582b055682c420e8633861baeb36a6d P df43785430814849be835b580aabb724: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 20:23:04.580626 8365 raft_consensus.cc:482] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:23:04.580847 8365 raft_consensus.cc:504] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:04.581621 8365 leader_election.cc:288] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0e64797c14e4e8e8f4bc6eb2463ae1f (127.31.235.130:39743), df43785430814849be835b580aabb724 (127.31.235.131:37035)
I20260620 20:23:04.582180 8106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b582b055682c420e8633861baeb36a6d" candidate_uuid: "db89954f95d646f48ec84241c5307e2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" is_pre_election: true
I20260620 20:23:04.582346 8250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b582b055682c420e8633861baeb36a6d" candidate_uuid: "db89954f95d646f48ec84241c5307e2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df43785430814849be835b580aabb724" is_pre_election: true
I20260620 20:23:04.582432 8106 raft_consensus.cc:2454] T b582b055682c420e8633861baeb36a6d P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db89954f95d646f48ec84241c5307e2e in term 0.
I20260620 20:23:04.582617 8250 raft_consensus.cc:2454] T b582b055682c420e8633861baeb36a6d P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db89954f95d646f48ec84241c5307e2e in term 0.
I20260620 20:23:04.583133 7900 leader_election.cc:302] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e64797c14e4e8e8f4bc6eb2463ae1f, db89954f95d646f48ec84241c5307e2e; no voters:
I20260620 20:23:04.583432 8365 raft_consensus.cc:2781] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:23:04.583577 8365 raft_consensus.cc:482] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:23:04.583683 8365 raft_consensus.cc:3037] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:04.587383 8365 raft_consensus.cc:504] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:04.588528 8106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b582b055682c420e8633861baeb36a6d" candidate_uuid: "db89954f95d646f48ec84241c5307e2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f"
I20260620 20:23:04.588558 8250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b582b055682c420e8633861baeb36a6d" candidate_uuid: "db89954f95d646f48ec84241c5307e2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df43785430814849be835b580aabb724"
I20260620 20:23:04.588735 8106 raft_consensus.cc:3037] T b582b055682c420e8633861baeb36a6d P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:04.588763 8250 raft_consensus.cc:3037] T b582b055682c420e8633861baeb36a6d P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:04.591718 8106 raft_consensus.cc:2454] T b582b055682c420e8633861baeb36a6d P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db89954f95d646f48ec84241c5307e2e in term 1.
I20260620 20:23:04.592126 8250 raft_consensus.cc:2454] T b582b055682c420e8633861baeb36a6d P df43785430814849be835b580aabb724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db89954f95d646f48ec84241c5307e2e in term 1.
I20260620 20:23:04.592247 7900 leader_election.cc:302] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e64797c14e4e8e8f4bc6eb2463ae1f, db89954f95d646f48ec84241c5307e2e; no voters:
I20260620 20:23:04.592552 8328 raft_consensus.cc:2781] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:23:04.592636 8365 leader_election.cc:288] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e [CANDIDATE]: Term 1 election: Requested vote from peers d0e64797c14e4e8e8f4bc6eb2463ae1f (127.31.235.130:39743), df43785430814849be835b580aabb724 (127.31.235.131:37035)
I20260620 20:23:04.592725 8328 raft_consensus.cc:686] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e [term 1 LEADER]: Becoming Leader. State: Replica: db89954f95d646f48ec84241c5307e2e, State: Running, Role: LEADER
I20260620 20:23:04.592976 8328 consensus_queue.cc:237] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:04.595932 7809 catalog_manager.cc:5495] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e reported cstate change: term changed from 0 to 1, leader changed from <none> to db89954f95d646f48ec84241c5307e2e (127.31.235.129). New cstate: current_term: 1 leader_uuid: "db89954f95d646f48ec84241c5307e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } health_report { overall_health: UNKNOWN } } }
I20260620 20:23:04.613701 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:04.607480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60296 (local address 127.31.235.129:35757)
0620 20:23:04.607664 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:04.607670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:04.607837 (+ 167us) server_negotiation.cc:407] Connection header received
0620 20:23:04.608069 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:04.608075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:04.608157 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:04.608244 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:23:04.608647 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.609872 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.611882 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:04.612254 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:04.612351 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:04.612914 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:23:04.613036 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:23:04.613105 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:04.613294 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 20:23:04.613369 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:23:04.616932 8106 raft_consensus.cc:1261] T b582b055682c420e8633861baeb36a6d P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 1 FOLLOWER]: Refusing update from remote peer db89954f95d646f48ec84241c5307e2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:23:04.617010 8250 raft_consensus.cc:1261] T b582b055682c420e8633861baeb36a6d P df43785430814849be835b580aabb724 [term 1 FOLLOWER]: Refusing update from remote peer db89954f95d646f48ec84241c5307e2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:23:04.617539 8387 consensus_queue.cc:1035] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:23:04.618045 8328 consensus_queue.cc:1035] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:23:04.626381 8373 mvcc.cc:205] Tried to move back new op lower bound from 7299018688985280512 to 7299018688895688704. Current Snapshot: MvccSnapshot[applied={T|T < 7299018688985280512}]
I20260620 20:23:04.644666 7809 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:23:04.644927 7809 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:23:04.648489 7809 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5: Sending DeleteTablet for 3 replicas of tablet b582b055682c420e8633861baeb36a6d
I20260620 20:23:04.649243 7947 tablet_service.cc:1514] Processing DeleteTablet for tablet b582b055682c420e8633861baeb36a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:23:04 UTC) from {username='oryx', principal='oryx/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:60266
I20260620 20:23:04.649557 8392 tablet_replica.cc:330] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e: stopping tablet replica
I20260620 20:23:04.649766 8392 raft_consensus.cc:2229] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:23:04.649793 8086 tablet_service.cc:1514] Processing DeleteTablet for tablet b582b055682c420e8633861baeb36a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:23:04 UTC) from {username='oryx', principal='oryx/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:53518
I20260620 20:23:04.650063 8394 tablet_replica.cc:330] T b582b055682c420e8633861baeb36a6d P d0e64797c14e4e8e8f4bc6eb2463ae1f: stopping tablet replica
I20260620 20:23:04.650249 8394 raft_consensus.cc:2229] T b582b055682c420e8633861baeb36a6d P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:23:04.650259 8392 raft_consensus.cc:2258] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:23:04.650271 8225 tablet_service.cc:1514] Processing DeleteTablet for tablet b582b055682c420e8633861baeb36a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:23:04 UTC) from {username='oryx', principal='oryx/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:57180
I20260620 20:23:04.650476 8394 raft_consensus.cc:2258] T b582b055682c420e8633861baeb36a6d P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:23:04.650514 8393 tablet_replica.cc:330] T b582b055682c420e8633861baeb36a6d P df43785430814849be835b580aabb724: stopping tablet replica
I20260620 20:23:04.650693 8393 raft_consensus.cc:2229] T b582b055682c420e8633861baeb36a6d P df43785430814849be835b580aabb724 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:23:04.650907 8393 raft_consensus.cc:2258] T b582b055682c420e8633861baeb36a6d P df43785430814849be835b580aabb724 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:23:04.651859 8394 ts_tablet_manager.cc:1905] T b582b055682c420e8633861baeb36a6d P d0e64797c14e4e8e8f4bc6eb2463ae1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:23:04.651871 8393 ts_tablet_manager.cc:1905] T b582b055682c420e8633861baeb36a6d P df43785430814849be835b580aabb724: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:23:04.652498 7810 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:23:04.653146 7810 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:23:04.657538 8392 ts_tablet_manager.cc:1905] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:23:04.658317 8393 ts_tablet_manager.cc:1918] T b582b055682c420e8633861baeb36a6d P df43785430814849be835b580aabb724: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:23:04.658456 8393 log.cc:1196] T b582b055682c420e8633861baeb36a6d P df43785430814849be835b580aabb724: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/b582b055682c420e8633861baeb36a6d
I20260620 20:23:04.658866 8393 ts_tablet_manager.cc:1939] T b582b055682c420e8633861baeb36a6d P df43785430814849be835b580aabb724: Deleting consensus metadata
I20260620 20:23:04.660655 7947 tablet_service.cc:1467] Processing CreateTablet for tablet dacddb3c9e6b43cfa9359550150204d7 (DEFAULT_TABLE table=test-table [id=4b229bc84c444308b30fb2d3173947df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:23:04.661101 7947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dacddb3c9e6b43cfa9359550150204d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:04.662148 8225 tablet_service.cc:1467] Processing CreateTablet for tablet dacddb3c9e6b43cfa9359550150204d7 (DEFAULT_TABLE table=test-table [id=4b229bc84c444308b30fb2d3173947df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:23:04.662736 8225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dacddb3c9e6b43cfa9359550150204d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:04.664721 8394 ts_tablet_manager.cc:1918] T b582b055682c420e8633861baeb36a6d P d0e64797c14e4e8e8f4bc6eb2463ae1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:23:04.665642 7794 catalog_manager.cc:4841] TS df43785430814849be835b580aabb724 (127.31.235.131:37035): tablet b582b055682c420e8633861baeb36a6d (table test-table [id=c932b6648413456488d1c52355296c0d]) successfully deleted
I20260620 20:23:04.667698 8392 ts_tablet_manager.cc:1918] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:23:04.666523 8086 tablet_service.cc:1467] Processing CreateTablet for tablet dacddb3c9e6b43cfa9359550150204d7 (DEFAULT_TABLE table=test-table [id=4b229bc84c444308b30fb2d3173947df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:23:04.668118 8086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dacddb3c9e6b43cfa9359550150204d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:04.674492 8411 tablet_bootstrap.cc:492] T dacddb3c9e6b43cfa9359550150204d7 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Bootstrap starting.
I20260620 20:23:04.674907 8394 log.cc:1196] T b582b055682c420e8633861baeb36a6d P d0e64797c14e4e8e8f4bc6eb2463ae1f: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/b582b055682c420e8633861baeb36a6d
I20260620 20:23:04.675469 8394 ts_tablet_manager.cc:1939] T b582b055682c420e8633861baeb36a6d P d0e64797c14e4e8e8f4bc6eb2463ae1f: Deleting consensus metadata
I20260620 20:23:04.676292 8411 tablet_bootstrap.cc:654] T dacddb3c9e6b43cfa9359550150204d7 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:04.676496 8392 log.cc:1196] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/b582b055682c420e8633861baeb36a6d
I20260620 20:23:04.677017 8392 ts_tablet_manager.cc:1939] T b582b055682c420e8633861baeb36a6d P db89954f95d646f48ec84241c5307e2e: Deleting consensus metadata
I20260620 20:23:04.683385 8410 tablet_bootstrap.cc:492] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724: Bootstrap starting.
I20260620 20:23:04.681074 8409 tablet_bootstrap.cc:492] T dacddb3c9e6b43cfa9359550150204d7 P db89954f95d646f48ec84241c5307e2e: Bootstrap starting.
I20260620 20:23:04.685238 8409 tablet_bootstrap.cc:654] T dacddb3c9e6b43cfa9359550150204d7 P db89954f95d646f48ec84241c5307e2e: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:04.685551 8410 tablet_bootstrap.cc:654] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:04.689893 8410 tablet_bootstrap.cc:492] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724: No bootstrap required, opened a new log
I20260620 20:23:04.690095 8410 ts_tablet_manager.cc:1397] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20260620 20:23:04.690748 7795 catalog_manager.cc:4841] TS db89954f95d646f48ec84241c5307e2e (127.31.235.129:35757): tablet b582b055682c420e8633861baeb36a6d (table test-table [id=c932b6648413456488d1c52355296c0d]) successfully deleted
I20260620 20:23:04.690830 8410 raft_consensus.cc:348] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:04.690954 7797 catalog_manager.cc:4841] TS d0e64797c14e4e8e8f4bc6eb2463ae1f (127.31.235.130:39743): tablet b582b055682c420e8633861baeb36a6d (table test-table [id=c932b6648413456488d1c52355296c0d]) successfully deleted
I20260620 20:23:04.691108 8410 raft_consensus.cc:374] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:04.691219 8410 raft_consensus.cc:729] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df43785430814849be835b580aabb724, State: Initialized, Role: FOLLOWER
I20260620 20:23:04.691489 8410 consensus_queue.cc:260] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:04.691993 8410 ts_tablet_manager.cc:1428] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 20:23:04.695432 8409 tablet_bootstrap.cc:492] T dacddb3c9e6b43cfa9359550150204d7 P db89954f95d646f48ec84241c5307e2e: No bootstrap required, opened a new log
I20260620 20:23:04.695597 8409 ts_tablet_manager.cc:1397] T dacddb3c9e6b43cfa9359550150204d7 P db89954f95d646f48ec84241c5307e2e: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.006s
I20260620 20:23:04.696350 8409 raft_consensus.cc:348] T dacddb3c9e6b43cfa9359550150204d7 P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:04.696599 8409 raft_consensus.cc:374] T dacddb3c9e6b43cfa9359550150204d7 P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:04.696676 8409 raft_consensus.cc:729] T dacddb3c9e6b43cfa9359550150204d7 P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db89954f95d646f48ec84241c5307e2e, State: Initialized, Role: FOLLOWER
I20260620 20:23:04.696832 8411 tablet_bootstrap.cc:492] T dacddb3c9e6b43cfa9359550150204d7 P d0e64797c14e4e8e8f4bc6eb2463ae1f: No bootstrap required, opened a new log
I20260620 20:23:04.697002 8411 ts_tablet_manager.cc:1397] T dacddb3c9e6b43cfa9359550150204d7 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.000s
I20260620 20:23:04.696950 8409 consensus_queue.cc:260] T dacddb3c9e6b43cfa9359550150204d7 P db89954f95d646f48ec84241c5307e2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:04.697759 8411 raft_consensus.cc:348] T dacddb3c9e6b43cfa9359550150204d7 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:04.698016 8411 raft_consensus.cc:374] T dacddb3c9e6b43cfa9359550150204d7 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:04.699561 8411 raft_consensus.cc:729] T dacddb3c9e6b43cfa9359550150204d7 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e64797c14e4e8e8f4bc6eb2463ae1f, State: Initialized, Role: FOLLOWER
I20260620 20:23:04.699887 8409 ts_tablet_manager.cc:1428] T dacddb3c9e6b43cfa9359550150204d7 P db89954f95d646f48ec84241c5307e2e: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260620 20:23:04.699901 8411 consensus_queue.cc:260] T dacddb3c9e6b43cfa9359550150204d7 P d0e64797c14e4e8e8f4bc6eb2463ae1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:04.702571 8411 ts_tablet_manager.cc:1428] T dacddb3c9e6b43cfa9359550150204d7 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 20:23:05.010569 8385 raft_consensus.cc:482] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:23:05.010851 8385 raft_consensus.cc:504] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:05.011540 8385 leader_election.cc:288] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db89954f95d646f48ec84241c5307e2e (127.31.235.129:35757), d0e64797c14e4e8e8f4bc6eb2463ae1f (127.31.235.130:39743)
I20260620 20:23:05.012156 8106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacddb3c9e6b43cfa9359550150204d7" candidate_uuid: "df43785430814849be835b580aabb724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" is_pre_election: true
I20260620 20:23:05.012174 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacddb3c9e6b43cfa9359550150204d7" candidate_uuid: "df43785430814849be835b580aabb724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db89954f95d646f48ec84241c5307e2e" is_pre_election: true
I20260620 20:23:05.012403 8106 raft_consensus.cc:2454] T dacddb3c9e6b43cfa9359550150204d7 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df43785430814849be835b580aabb724 in term 0.
I20260620 20:23:05.012428 7970 raft_consensus.cc:2454] T dacddb3c9e6b43cfa9359550150204d7 P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df43785430814849be835b580aabb724 in term 0.
I20260620 20:23:05.012907 8179 leader_election.cc:302] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0e64797c14e4e8e8f4bc6eb2463ae1f, df43785430814849be835b580aabb724; no voters:
I20260620 20:23:05.013325 8385 raft_consensus.cc:2781] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:23:05.013440 8385 raft_consensus.cc:482] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:23:05.013588 8385 raft_consensus.cc:3037] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:05.015888 8385 raft_consensus.cc:504] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:05.016489 8385 leader_election.cc:288] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 [CANDIDATE]: Term 1 election: Requested vote from peers db89954f95d646f48ec84241c5307e2e (127.31.235.129:35757), d0e64797c14e4e8e8f4bc6eb2463ae1f (127.31.235.130:39743)
I20260620 20:23:05.016948 7970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacddb3c9e6b43cfa9359550150204d7" candidate_uuid: "df43785430814849be835b580aabb724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db89954f95d646f48ec84241c5307e2e"
I20260620 20:23:05.017021 8106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dacddb3c9e6b43cfa9359550150204d7" candidate_uuid: "df43785430814849be835b580aabb724" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f"
I20260620 20:23:05.017202 7970 raft_consensus.cc:3037] T dacddb3c9e6b43cfa9359550150204d7 P db89954f95d646f48ec84241c5307e2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:05.017226 8106 raft_consensus.cc:3037] T dacddb3c9e6b43cfa9359550150204d7 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:05.020155 7970 raft_consensus.cc:2454] T dacddb3c9e6b43cfa9359550150204d7 P db89954f95d646f48ec84241c5307e2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df43785430814849be835b580aabb724 in term 1.
I20260620 20:23:05.020166 8106 raft_consensus.cc:2454] T dacddb3c9e6b43cfa9359550150204d7 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df43785430814849be835b580aabb724 in term 1.
I20260620 20:23:05.020570 8177 leader_election.cc:302] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db89954f95d646f48ec84241c5307e2e, df43785430814849be835b580aabb724; no voters:
I20260620 20:23:05.020927 8385 raft_consensus.cc:2781] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:23:05.021404 8385 raft_consensus.cc:686] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 [term 1 LEADER]: Becoming Leader. State: Replica: df43785430814849be835b580aabb724, State: Running, Role: LEADER
I20260620 20:23:05.021709 8385 consensus_queue.cc:237] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } }
I20260620 20:23:05.024435 7810 catalog_manager.cc:5495] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 reported cstate change: term changed from 0 to 1, leader changed from <none> to df43785430814849be835b580aabb724 (127.31.235.131). New cstate: current_term: 1 leader_uuid: "df43785430814849be835b580aabb724" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df43785430814849be835b580aabb724" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 37035 } health_report { overall_health: HEALTHY } } }
I20260620 20:23:05.064958 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:05.059849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.31.235.131:37035)
0620 20:23:05.060411 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:05.060415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:05.060431 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:23:05.060482 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:05.060486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:05.060543 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:05.060630 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:23:05.061027 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:05.061875 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:05.063212 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:05.063645 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:05.063753 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:05.064198 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:23:05.064306 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:23:05.064400 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:05.064506 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 20:23:05.064564 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":335,"threads_started":1}
I20260620 20:23:05.074540 8106 raft_consensus.cc:1261] T dacddb3c9e6b43cfa9359550150204d7 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 1 FOLLOWER]: Refusing update from remote peer df43785430814849be835b580aabb724: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:23:05.074829 7970 raft_consensus.cc:1261] T dacddb3c9e6b43cfa9359550150204d7 P db89954f95d646f48ec84241c5307e2e [term 1 FOLLOWER]: Refusing update from remote peer df43785430814849be835b580aabb724: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:23:05.075115 8385 consensus_queue.cc:1035] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e64797c14e4e8e8f4bc6eb2463ae1f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:23:05.075429 8422 consensus_queue.cc:1035] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db89954f95d646f48ec84241c5307e2e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:23:05.118024 7810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:23:05.118242 7810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:23:05.121049 7810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01679f23407f4a689678b3c4bd7ba0f5: Sending DeleteTablet for 3 replicas of tablet dacddb3c9e6b43cfa9359550150204d7
I20260620 20:23:05.121868 7947 tablet_service.cc:1514] Processing DeleteTablet for tablet dacddb3c9e6b43cfa9359550150204d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:23:05 UTC) from {username='oryx', principal='oryx/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:60266
I20260620 20:23:05.121922 8086 tablet_service.cc:1514] Processing DeleteTablet for tablet dacddb3c9e6b43cfa9359550150204d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:23:05 UTC) from {username='oryx', principal='oryx/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:53518
I20260620 20:23:05.122186 8225 tablet_service.cc:1514] Processing DeleteTablet for tablet dacddb3c9e6b43cfa9359550150204d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:23:05 UTC) from {username='oryx', principal='oryx/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:57180
I20260620 20:23:05.122256 8392 tablet_replica.cc:330] T dacddb3c9e6b43cfa9359550150204d7 P db89954f95d646f48ec84241c5307e2e: stopping tablet replica
I20260620 20:23:05.122455 8393 tablet_replica.cc:330] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724: stopping tablet replica
I20260620 20:23:05.122471 8392 raft_consensus.cc:2229] T dacddb3c9e6b43cfa9359550150204d7 P db89954f95d646f48ec84241c5307e2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:23:05.122610 8393 raft_consensus.cc:2229] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:23:05.122689 8392 raft_consensus.cc:2258] T dacddb3c9e6b43cfa9359550150204d7 P db89954f95d646f48ec84241c5307e2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:23:05.122970 8394 tablet_replica.cc:330] T dacddb3c9e6b43cfa9359550150204d7 P d0e64797c14e4e8e8f4bc6eb2463ae1f: stopping tablet replica
I20260620 20:23:05.122962 8393 raft_consensus.cc:2258] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:23:05.123157 8394 raft_consensus.cc:2229] T dacddb3c9e6b43cfa9359550150204d7 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:23:05.123396 8394 raft_consensus.cc:2258] T dacddb3c9e6b43cfa9359550150204d7 P d0e64797c14e4e8e8f4bc6eb2463ae1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:23:05.124012 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 7881
I20260620 20:23:05.124059 8393 ts_tablet_manager.cc:1905] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:23:05.124609 8394 ts_tablet_manager.cc:1905] T dacddb3c9e6b43cfa9359550150204d7 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:23:05.128799 8392 ts_tablet_manager.cc:1905] T dacddb3c9e6b43cfa9359550150204d7 P db89954f95d646f48ec84241c5307e2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:23:05.130483 8393 ts_tablet_manager.cc:1918] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:23:05.130610 8393 log.cc:1196] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/dacddb3c9e6b43cfa9359550150204d7
I20260620 20:23:05.131016 8393 ts_tablet_manager.cc:1939] T dacddb3c9e6b43cfa9359550150204d7 P df43785430814849be835b580aabb724: Deleting consensus metadata
I20260620 20:23:05.132025 7794 catalog_manager.cc:4841] TS df43785430814849be835b580aabb724 (127.31.235.131:37035): tablet dacddb3c9e6b43cfa9359550150204d7 (table test-table [id=4b229bc84c444308b30fb2d3173947df]) successfully deleted
I20260620 20:23:05.133064 8394 ts_tablet_manager.cc:1918] T dacddb3c9e6b43cfa9359550150204d7 P d0e64797c14e4e8e8f4bc6eb2463ae1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:23:05.133212 8394 log.cc:1196] T dacddb3c9e6b43cfa9359550150204d7 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/dacddb3c9e6b43cfa9359550150204d7
I20260620 20:23:05.133741 8394 ts_tablet_manager.cc:1939] T dacddb3c9e6b43cfa9359550150204d7 P d0e64797c14e4e8e8f4bc6eb2463ae1f: Deleting consensus metadata
I20260620 20:23:05.134913 7797 catalog_manager.cc:4841] TS d0e64797c14e4e8e8f4bc6eb2463ae1f (127.31.235.130:39743): tablet dacddb3c9e6b43cfa9359550150204d7 (table test-table [id=4b229bc84c444308b30fb2d3173947df]) successfully deleted
I20260620 20:23:05.135408 8392 ts_tablet_manager.cc:1918] T dacddb3c9e6b43cfa9359550150204d7 P db89954f95d646f48ec84241c5307e2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 20:23:05.135505 8392 log.cc:1196] T dacddb3c9e6b43cfa9359550150204d7 P db89954f95d646f48ec84241c5307e2e: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/dacddb3c9e6b43cfa9359550150204d7
I20260620 20:23:05.135807 8392 ts_tablet_manager.cc:1939] T dacddb3c9e6b43cfa9359550150204d7 P db89954f95d646f48ec84241c5307e2e: Deleting consensus metadata
I20260620 20:23:05.136610 7795 catalog_manager.cc:4841] TS db89954f95d646f48ec84241c5307e2e (127.31.235.129:35757): tablet dacddb3c9e6b43cfa9359550150204d7 (table test-table [id=4b229bc84c444308b30fb2d3173947df]) successfully deleted
Jun 20 20:23:05 dist-test-slave-zzwf krb5kdc[7765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.235.129@KRBTEST.COM
I20260620 20:23:05.160840 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:05.126180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60312 (local address 127.31.235.129:35757)
0620 20:23:05.142941 (+ 16761us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:05.142947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:05.142972 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:23:05.143033 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:05.143038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:05.143125 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:05.143280 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:05.144720 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:05.146073 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:05.148083 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:05.148787 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:05.148908 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:05.154345 (+ 5437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:05.154367 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:05.154389 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:05.154428 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:05.158055 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:05.158873 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:05.158880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:05.158882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:05.159012 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:05.159435 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:05.159440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:05.159442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:05.159797 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:05.160018 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:05.160350 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 20:23:05.160515 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16632,"thread_start_us":17201,"threads_started":1}
I20260620 20:23:05.161538 8012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:23:05.308290 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 7881
I20260620 20:23:05.333778 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 8021
Jun 20 20:23:05 dist-test-slave-zzwf krb5kdc[7765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.235.130@KRBTEST.COM
I20260620 20:23:05.348932 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:05.334564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53540 (local address 127.31.235.130:39743)
0620 20:23:05.335037 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:05.335042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:05.335056 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:23:05.335109 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:05.335113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:05.335186 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:05.335310 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:05.336506 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:05.337247 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:05.338671 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:05.339132 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:05.339277 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:05.343317 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:05.343331 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:05.343347 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:05.343380 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:05.346287 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:05.346946 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:05.346955 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:05.346957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:05.347093 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:05.347442 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:05.347449 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:05.347452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:05.347786 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:05.348062 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:05.348358 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 20:23:05.348568 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":231,"threads_started":1}
I20260620 20:23:05.349720 8151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:23:05.474869 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 8021
I20260620 20:23:05.497139 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 8160
Jun 20 20:23:05 dist-test-slave-zzwf krb5kdc[7765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.235.131@KRBTEST.COM
I20260620 20:23:05.512027 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:05.497926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57212 (local address 127.31.235.131:37035)
0620 20:23:05.498108 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:05.498113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:05.498130 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:23:05.498350 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:05.498354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:05.498420 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:05.498517 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:05.499751 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:05.500494 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:05.501877 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:05.502355 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:05.502451 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:05.506685 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:05.506699 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:05.506716 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:05.506754 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:05.509708 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:05.510376 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:05.510387 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:05.510392 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:05.510526 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:05.510863 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:05.510866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:05.510868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:05.511189 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:05.511402 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:05.511620 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 20:23:05.511785 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:23:05.512646 8290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:23:05.635286 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 8160
I20260620 20:23:05.656970 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 7778
I20260620 20:23:05.658188 7850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:23:05.771767 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 7778
2026-06-20T20:23:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6071 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:23:05 dist-test-slave-zzwf krb5kdc[8440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:23:05 dist-test-slave-zzwf krb5kdc[8440](info): set up 2 sockets
Jun 20 20:23:05 dist-test-slave-zzwf krb5kdc[8440](info): commencing operation
krb5kdc: starting...
W20260620 20:23:06.867519 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:23:07 dist-test-slave-zzwf krb5kdc[8440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:23:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:23:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:23:07.097904 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:36347
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:36347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:23:07.423792 8453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:07.424122 8453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:07.424304 8453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:07.431614 8453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:23:07.431754 8453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:07.431838 8453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:07.431926 8453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:23:07.432019 8453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:23:07.441992 8453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:36347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:36347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:07.443310 8453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:07.444620 8453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:07.456180 8461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:07.456151 8458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:07.456127 8459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:07.457435 8453 server_base.cc:1034] running on GCE node
I20260620 20:23:07.458226 8453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:07.459697 8453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:07.460930 8453 hybrid_clock.cc:648] HybridClock initialized: now 1781986987460890 us; error 50 us; skew 500 ppm
I20260620 20:23:07.461268 8453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T20:23:07Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1757 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:23:07 dist-test-slave-zzwf krb5kdc[8467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:23:07 dist-test-slave-zzwf krb5kdc[8467](info): set up 2 sockets
Jun 20 20:23:07 dist-test-slave-zzwf krb5kdc[8467](info): commencing operation
krb5kdc: starting...
W20260620 20:23:08.662822 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:23:08 dist-test-slave-zzwf krb5kdc[8467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:23:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:23:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:23:08.889667 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:35007
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:35007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:23:09.210692 8480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:09.211054 8480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:09.211237 8480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:09.218569 8480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:23:09.218719 8480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:09.218827 8480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:09.219182 8480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:23:09.219305 8480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:23:09.229266 8480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:35007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:35007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:09.230643 8480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:09.231942 8480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:09.243661 8488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:09.243615 8486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:09.243711 8485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:09.244828 8480 server_base.cc:1034] running on GCE node
I20260620 20:23:09.245621 8480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:09.246940 8480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:09.248138 8480 hybrid_clock.cc:648] HybridClock initialized: now 1781986989248095 us; error 34 us; skew 500 ppm
I20260620 20:23:09.248456 8480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:23:09 dist-test-slave-zzwf krb5kdc[8467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986989, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:23:09.251446 8480 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:23:09.252805 8480 webserver.cc:460] Webserver started at http://127.31.235.190:42579/ using document root <none> and password file <none>
I20260620 20:23:09.253453 8480 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:09.253616 8480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:09.253928 8480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:23:09.256707 8480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "015a8f0a189e40afabd09bf719a17d2b"
format_stamp: "Formatted at 2026-06-20 20:23:09 on dist-test-slave-zzwf"
server_key: "b4a51e17872373c403b5dc147631aed8"
server_key_iv: "c4167f992629b2ef357c4a54608beb49"
server_key_version: "encryptionkey@0"
I20260620 20:23:09.257401 8480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "015a8f0a189e40afabd09bf719a17d2b"
format_stamp: "Formatted at 2026-06-20 20:23:09 on dist-test-slave-zzwf"
server_key: "b4a51e17872373c403b5dc147631aed8"
server_key_iv: "c4167f992629b2ef357c4a54608beb49"
server_key_version: "encryptionkey@0"
I20260620 20:23:09.262668 8480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:23:09.266007 8495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:09.267319 8480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:23:09.267462 8480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "015a8f0a189e40afabd09bf719a17d2b"
format_stamp: "Formatted at 2026-06-20 20:23:09 on dist-test-slave-zzwf"
server_key: "b4a51e17872373c403b5dc147631aed8"
server_key_iv: "c4167f992629b2ef357c4a54608beb49"
server_key_version: "encryptionkey@0"
I20260620 20:23:09.267619 8480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:09.300774 8480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:09.304836 8480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:09.305075 8480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:09.323746 8480 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:35007
I20260620 20:23:09.323755 8546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:35007 every 8 connection(s)
I20260620 20:23:09.325249 8480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:23:09.329293 8547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:09.331578 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 8480
I20260620 20:23:09.331809 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:23:09.332677 32686 external_mini_cluster.cc:1428] Setting key 9e8f343dad0959ee299ff63e5c1b84f2
I20260620 20:23:09.342558 8547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b: Bootstrap starting.
I20260620 20:23:09.347054 8547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:09.348464 8547 log.cc:824] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:23:09 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:23:09.352233 8547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b: No bootstrap required, opened a new log
I20260620 20:23:09.360981 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:09.335022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.31.235.190:35007)
0620 20:23:09.335950 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:09.335963 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:09.336012 (+ 49us) server_negotiation.cc:407] Connection header received
0620 20:23:09.338059 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:09.338082 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:09.338498 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:09.338946 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:09.340510 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:09.342066 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:09.344433 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:09.345689 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:09.345876 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:09.351943 (+ 6067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:09.351971 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:09.351987 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:09.352035 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:09.356516 (+ 4481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:09.357444 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:09.357453 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:09.357467 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:09.357797 (+ 330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:09.358246 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:09.358250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:09.358252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:09.358831 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:09.359045 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:09.359519 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 20:23:09.359914 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":402,"threads_started":1}
I20260620 20:23:09.362105 8547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015a8f0a189e40afabd09bf719a17d2b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35007 } }
I20260620 20:23:09.362488 8547 raft_consensus.cc:374] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:09.362607 8547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 015a8f0a189e40afabd09bf719a17d2b, State: Initialized, Role: FOLLOWER
I20260620 20:23:09.363220 8547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015a8f0a189e40afabd09bf719a17d2b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35007 } }
I20260620 20:23:09.363476 8547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:23:09.363632 8547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:23:09.363808 8547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:09.365940 8547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015a8f0a189e40afabd09bf719a17d2b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35007 } }
I20260620 20:23:09.366367 8547 leader_election.cc:302] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 015a8f0a189e40afabd09bf719a17d2b; no voters:
I20260620 20:23:09.366861 8547 leader_election.cc:288] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:23:09.367077 8552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:23:09.367897 8552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [term 1 LEADER]: Becoming Leader. State: Replica: 015a8f0a189e40afabd09bf719a17d2b, State: Running, Role: LEADER
I20260620 20:23:09.368485 8552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015a8f0a189e40afabd09bf719a17d2b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35007 } }
I20260620 20:23:09.369002 8547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:23:09.372527 8554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 015a8f0a189e40afabd09bf719a17d2b. Latest consensus state: current_term: 1 leader_uuid: "015a8f0a189e40afabd09bf719a17d2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015a8f0a189e40afabd09bf719a17d2b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35007 } } }
I20260620 20:23:09.372936 8554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [sys.catalog]: This master's current role is: LEADER
I20260620 20:23:09.374184 8553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "015a8f0a189e40afabd09bf719a17d2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015a8f0a189e40afabd09bf719a17d2b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35007 } } }
I20260620 20:23:09.374497 8553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [sys.catalog]: This master's current role is: LEADER
I20260620 20:23:09.375355 8559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:23:09.379968 8559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:23:09.389745 8559 catalog_manager.cc:1347] Generated new cluster ID: 0c7a9df4587540389fe02d549f17530d
I20260620 20:23:09.389885 8559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:23:09.406991 8559 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:23:09.408300 8559 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:23:09.413386 8559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b: Generated new TSK 0
I20260620 20:23:09.414374 8559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
I20260620 20:23:09.481475 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:35007
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:23:09.806306 8573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:09.806581 8573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:09.806756 8573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:09.813875 8573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:09.813974 8573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:09.814174 8573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:23:09.823556 8573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:35007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:09.824802 8573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:09.826097 8573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:09.838735 8578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:09.838698 8581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:09.838774 8579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:09.839670 8573 server_base.cc:1034] running on GCE node
I20260620 20:23:09.840168 8573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:09.840998 8573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:09.842211 8573 hybrid_clock.cc:648] HybridClock initialized: now 1781986989842129 us; error 54 us; skew 500 ppm
I20260620 20:23:09.842528 8573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:23:09 dist-test-slave-zzwf krb5kdc[8467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986989, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:23:09.845909 8573 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:23:09.847333 8573 webserver.cc:460] Webserver started at http://127.31.235.129:37421/ using document root <none> and password file <none>
I20260620 20:23:09.847981 8573 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:09.848124 8573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:09.848433 8573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:23:09.851387 8573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "b8df0be8c2864b109103c92ce553c565"
format_stamp: "Formatted at 2026-06-20 20:23:09 on dist-test-slave-zzwf"
server_key: "e4096ad409981668c992b8d5eb4dc578"
server_key_iv: "c9504bcf185416289930b878f95d5b41"
server_key_version: "encryptionkey@0"
I20260620 20:23:09.852149 8573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "b8df0be8c2864b109103c92ce553c565"
format_stamp: "Formatted at 2026-06-20 20:23:09 on dist-test-slave-zzwf"
server_key: "e4096ad409981668c992b8d5eb4dc578"
server_key_iv: "c9504bcf185416289930b878f95d5b41"
server_key_version: "encryptionkey@0"
I20260620 20:23:09.856946 8573 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 20:23:09.860374 8588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:09.861765 8573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 20:23:09.861932 8573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "b8df0be8c2864b109103c92ce553c565"
format_stamp: "Formatted at 2026-06-20 20:23:09 on dist-test-slave-zzwf"
server_key: "e4096ad409981668c992b8d5eb4dc578"
server_key_iv: "c9504bcf185416289930b878f95d5b41"
server_key_version: "encryptionkey@0"
I20260620 20:23:09.862123 8573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:09.884325 8573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:09.888605 8573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:09.888921 8573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:09.889886 8573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:23:09.891144 8573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:23:09.891256 8573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:09.891397 8573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:23:09.891469 8573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:09.924131 8573 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:40821
I20260620 20:23:09.924218 8700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:40821 every 8 connection(s)
I20260620 20:23:09.925402 8573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:23:09.933004 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 8573
I20260620 20:23:09.933207 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:23:09.934007 32686 external_mini_cluster.cc:1428] Setting key ce2340fe23b23c42e3b892ffc167ef52
Jun 20 20:23:09 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986989, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:23:09.945761 8703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:09.927927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:54331 (local address 127.31.235.190:35007)
0620 20:23:09.928347 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:09.928351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:09.929270 (+ 919us) server_negotiation.cc:407] Connection header received
0620 20:23:09.931019 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:09.931023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:09.931086 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:09.931188 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:09.932584 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:09.933758 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:09.935689 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:09.936229 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:09.936365 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:09.940027 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:09.940032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:09.940035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:09.940064 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:09.942507 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:09.943377 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:09.943385 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:09.943391 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:09.943528 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:09.943899 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:09.943906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:09.943913 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:09.944271 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:09.944416 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:09.945018 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 20:23:09.945270 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":202,"threads_started":1}
I20260620 20:23:09.946406 8704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:09.928451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.129:54331)
0620 20:23:09.929118 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:09.929153 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:09.930782 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:09.931405 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:09.931412 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:09.931869 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:09.932396 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:09.932413 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:09.933954 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:09.933957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:09.935551 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:09.935564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:09.936439 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:09.936444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:09.936636 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:09.937308 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:09.937335 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:09.939789 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:23:09.942641 (+ 2852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:09.942652 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:09.942668 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:09.943262 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:09.943630 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:09.943634 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:09.943636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:09.943798 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:09.944384 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:23:09.944389 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:23:09.944669 (+ 280us) client_negotiation.cc:769] Sending connection context
0620 20:23:09.944886 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 20:23:09.945237 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":270,"threads_started":1}
I20260620 20:23:09.947921 8701 heartbeater.cc:347] Connected to a master server at 127.31.235.190:35007
I20260620 20:23:09.948227 8701 heartbeater.cc:464] Registering TS with master...
I20260620 20:23:09.949054 8701 heartbeater.cc:511] Master 127.31.235.190:35007 requested a full tablet report, sending...
I20260620 20:23:09.951215 8512 ts_manager.cc:194] Registered new tserver with Master: b8df0be8c2864b109103c92ce553c565 (127.31.235.129:40821)
I20260620 20:23:09.952733 8512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:54331
WARNING: no policy specified for kudu/127.31.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
I20260620 20:23:09.985338 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:35007
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:23:10.316808 8707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:10.317139 8707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:10.317308 8707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:10.324625 8707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:10.324774 8707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:10.324944 8707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:23:10.334615 8707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:35007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:10.335790 8707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:10.337034 8707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:10.349720 8715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:10.349834 8712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:10.349799 8713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:10.350821 8707 server_base.cc:1034] running on GCE node
I20260620 20:23:10.351454 8707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:10.352507 8707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:10.353749 8707 hybrid_clock.cc:648] HybridClock initialized: now 1781986990353697 us; error 43 us; skew 500 ppm
I20260620 20:23:10.354077 8707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:23:10 dist-test-slave-zzwf krb5kdc[8467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986990, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:23:10.357163 8707 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:23:10.358665 8707 webserver.cc:460] Webserver started at http://127.31.235.130:37615/ using document root <none> and password file <none>
I20260620 20:23:10.359330 8707 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:10.359455 8707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:10.359773 8707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:23:10.362620 8707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "a627dd6264334eb790f294b1c34de6c2"
format_stamp: "Formatted at 2026-06-20 20:23:10 on dist-test-slave-zzwf"
server_key: "f197cf766f85c527f9a2d8ee68d55782"
server_key_iv: "c31349c1ddaa046228d8a46e8591d3e2"
server_key_version: "encryptionkey@0"
I20260620 20:23:10.363402 8707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "a627dd6264334eb790f294b1c34de6c2"
format_stamp: "Formatted at 2026-06-20 20:23:10 on dist-test-slave-zzwf"
server_key: "f197cf766f85c527f9a2d8ee68d55782"
server_key_iv: "c31349c1ddaa046228d8a46e8591d3e2"
server_key_version: "encryptionkey@0"
I20260620 20:23:10.368528 8707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:23:10.372092 8722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:10.373445 8707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 20:23:10.373638 8707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "a627dd6264334eb790f294b1c34de6c2"
format_stamp: "Formatted at 2026-06-20 20:23:10 on dist-test-slave-zzwf"
server_key: "f197cf766f85c527f9a2d8ee68d55782"
server_key_iv: "c31349c1ddaa046228d8a46e8591d3e2"
server_key_version: "encryptionkey@0"
I20260620 20:23:10.373826 8707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:10.396930 8707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:10.401885 8707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:10.402181 8707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:10.403195 8707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:23:10.404560 8707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:23:10.404631 8707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:10.404775 8707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:23:10.404824 8707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:10.438103 8707 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:34993
I20260620 20:23:10.438190 8834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:34993 every 8 connection(s)
I20260620 20:23:10.439347 8707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:23:10.447611 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 8707
I20260620 20:23:10.447800 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:23:10.448268 32686 external_mini_cluster.cc:1428] Setting key dbbde55c45afef0dd388f2c442ff7da8
Jun 20 20:23:10 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986990, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:23:10.459674 8703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:10.441989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:48717 (local address 127.31.235.190:35007)
0620 20:23:10.442163 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:10.442168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:10.443216 (+ 1048us) server_negotiation.cc:407] Connection header received
0620 20:23:10.444982 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:10.444991 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:10.445079 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:10.445199 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:10.446656 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:10.447351 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:10.449246 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:10.449766 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:10.449902 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:10.453998 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:10.454003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:10.454007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:10.454038 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:10.456664 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:10.457392 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:10.457396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:10.457398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:10.457574 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:10.457984 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:10.457992 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:10.457995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:10.458264 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:10.458427 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:10.459121 (+ 694us) server_negotiation.cc:299] Negotiation successful
0620 20:23:10.459337 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:23:10.460683 8837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:10.442392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.130:48717)
0620 20:23:10.443059 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:10.443101 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:10.444717 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:10.445419 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:10.445426 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:10.445965 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:10.446494 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:10.446510 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:10.447517 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:10.447521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:10.449094 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:10.449107 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:10.449984 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:10.449991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:10.450321 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:10.451007 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:10.451042 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:10.453761 (+ 2719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:23:10.456808 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:10.456817 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:10.456834 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:10.457252 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:10.457688 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:10.457693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:10.457697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:10.457873 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:10.458387 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:23:10.458395 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:23:10.458747 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 20:23:10.459029 (+ 282us) client_negotiation.cc:240] Negotiation successful
0620 20:23:10.459553 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":260,"threads_started":1}
I20260620 20:23:10.462299 8835 heartbeater.cc:347] Connected to a master server at 127.31.235.190:35007
I20260620 20:23:10.462628 8835 heartbeater.cc:464] Registering TS with master...
I20260620 20:23:10.463325 8835 heartbeater.cc:511] Master 127.31.235.190:35007 requested a full tablet report, sending...
I20260620 20:23:10.464805 8512 ts_manager.cc:194] Registered new tserver with Master: a627dd6264334eb790f294b1c34de6c2 (127.31.235.130:34993)
I20260620 20:23:10.465723 8512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:48717
WARNING: no policy specified for kudu/127.31.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
I20260620 20:23:10.497648 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:35007
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:23:10.823629 8840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:10.823904 8840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:10.824061 8840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:10.831291 8840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:10.831401 8840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:10.831576 8840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:23:10.841427 8840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:35007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:10.842765 8840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:10.844024 8840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:10.857110 8846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:10.857369 8848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:10.857604 8845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:10.857919 8840 server_base.cc:1034] running on GCE node
I20260620 20:23:10.858457 8840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:10.859380 8840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:10.860574 8840 hybrid_clock.cc:648] HybridClock initialized: now 1781986990860533 us; error 32 us; skew 500 ppm
I20260620 20:23:10.860849 8840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:23:10 dist-test-slave-zzwf krb5kdc[8467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986990, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:23:10.864001 8840 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:23:10.865684 8840 webserver.cc:460] Webserver started at http://127.31.235.131:45575/ using document root <none> and password file <none>
I20260620 20:23:10.866457 8840 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:10.866607 8840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:10.866946 8840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:23:10.870172 8840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "2ca1c598c07144c985728a9e320cfe88"
format_stamp: "Formatted at 2026-06-20 20:23:10 on dist-test-slave-zzwf"
server_key: "b3c0e8b6f7b8e482291ba75576305243"
server_key_iv: "0f9ecc2cfe4c638ea5be3b535a7f6b3e"
server_key_version: "encryptionkey@0"
I20260620 20:23:10.870951 8840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "2ca1c598c07144c985728a9e320cfe88"
format_stamp: "Formatted at 2026-06-20 20:23:10 on dist-test-slave-zzwf"
server_key: "b3c0e8b6f7b8e482291ba75576305243"
server_key_iv: "0f9ecc2cfe4c638ea5be3b535a7f6b3e"
server_key_version: "encryptionkey@0"
I20260620 20:23:10.876416 8840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 20:23:10.880069 8855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:10.881870 8840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:23:10.882061 8840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "2ca1c598c07144c985728a9e320cfe88"
format_stamp: "Formatted at 2026-06-20 20:23:10 on dist-test-slave-zzwf"
server_key: "b3c0e8b6f7b8e482291ba75576305243"
server_key_iv: "0f9ecc2cfe4c638ea5be3b535a7f6b3e"
server_key_version: "encryptionkey@0"
I20260620 20:23:10.882344 8840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:10.909789 8840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:10.914422 8840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:10.914695 8840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:10.915660 8840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:23:10.917062 8840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:23:10.917155 8840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:10.917248 8840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:23:10.917297 8840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:10.953178 8840 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:36943
I20260620 20:23:10.953269 8967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:36943 every 8 connection(s)
I20260620 20:23:10.954488 8840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:23:10.956319 8701 heartbeater.cc:503] Master 127.31.235.190:35007 was elected leader, sending a full tablet report...
I20260620 20:23:10.958647 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 8840
I20260620 20:23:10.958839 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:23:10.959270 32686 external_mini_cluster.cc:1428] Setting key 99eac29cdd92cea803318d7f5c1a7869
Jun 20 20:23:10 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986990, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:23:10.974370 8703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:10.957176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:52975 (local address 127.31.235.190:35007)
0620 20:23:10.957365 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:10.957369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:10.958645 (+ 1276us) server_negotiation.cc:407] Connection header received
0620 20:23:10.960592 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:10.960597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:10.960706 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:10.960787 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:10.962569 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:10.963295 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:10.964744 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:10.965182 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:10.965289 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:10.968507 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:10.968512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:10.968515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:10.968544 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:10.970994 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:10.971784 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:10.971789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:10.971791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:10.971930 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:10.972398 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:10.972401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:10.972403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:10.972689 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:10.972861 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:10.973764 (+ 903us) server_negotiation.cc:299] Negotiation successful
0620 20:23:10.973995 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:23:10.975313 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:10.957643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.131:52975)
0620 20:23:10.958474 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:10.958518 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:10.960307 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:10.961062 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:10.961070 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:10.961722 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:10.962427 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:10.962446 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:10.963437 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:10.963441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:10.964612 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:10.964624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:10.965377 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:10.965382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:10.965656 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:10.966245 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:10.966290 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:10.968341 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:23:10.971191 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:10.971206 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:10.971219 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:10.971646 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:10.972031 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:10.972034 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:10.972037 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:10.972271 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:10.972849 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:23:10.972855 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:23:10.973296 (+ 441us) client_negotiation.cc:769] Sending connection context
0620 20:23:10.973617 (+ 321us) client_negotiation.cc:240] Negotiation successful
0620 20:23:10.974157 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":348,"threads_started":1}
I20260620 20:23:10.976601 8968 heartbeater.cc:347] Connected to a master server at 127.31.235.190:35007
I20260620 20:23:10.976933 8968 heartbeater.cc:464] Registering TS with master...
I20260620 20:23:10.977690 8968 heartbeater.cc:511] Master 127.31.235.190:35007 requested a full tablet report, sending...
I20260620 20:23:10.979012 8512 ts_manager.cc:194] Registered new tserver with Master: 2ca1c598c07144c985728a9e320cfe88 (127.31.235.131:36943)
I20260620 20:23:10.979791 8512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:52975
I20260620 20:23:10.986902 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.129.keytab.
Entry for principal oryx/127.31.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.129.keytab.
WARNING: no policy specified for oryx/127.31.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.130.keytab.
Entry for principal oryx/127.31.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.130.keytab.
WARNING: no policy specified for oryx/127.31.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.131.keytab.
Entry for principal oryx/127.31.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.131.keytab.
I20260620 20:23:11.134001 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 8573
Jun 20 20:23:11 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:23:11.149894 8978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:11.135280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46936 (local address 127.31.235.129:40821)
0620 20:23:11.135974 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:11.135984 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:11.136012 (+ 28us) server_negotiation.cc:407] Connection header received
0620 20:23:11.136072 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:11.136078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:11.136307 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:11.136447 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:11.137761 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:11.139064 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:11.140257 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:11.140818 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:11.140917 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:11.144783 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:11.144793 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:11.144807 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:11.144836 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:11.147589 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:11.148223 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:11.148230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:11.148232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:11.148352 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:11.148642 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:11.148645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:11.148647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:11.148956 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:11.149184 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:11.149362 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 20:23:11.149605 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":392,"threads_started":1}
I20260620 20:23:11.151165 8696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:23:11.233142 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 8573
I20260620 20:23:11.249428 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 8707
Jun 20 20:23:11 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.130@KRBTEST.COM
I20260620 20:23:11.265101 8980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:11.250234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.31.235.130:34993)
0620 20:23:11.250742 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:11.250749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:11.250773 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:23:11.250838 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:11.250844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:11.251053 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:11.251256 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:11.252475 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:11.253443 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:11.254767 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:11.255371 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:11.255511 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:11.259615 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:11.259626 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:11.259641 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:11.259679 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:11.262443 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:11.263064 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:11.263071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:11.263073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:11.263196 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:11.263551 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:11.263557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:11.263560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:11.263895 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:11.264190 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:11.264495 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 20:23:11.264767 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":260,"threads_started":1}
I20260620 20:23:11.266402 8830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:23:11.352982 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 8707
I20260620 20:23:11.368625 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 8840
Jun 20 20:23:11 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.131@KRBTEST.COM
I20260620 20:23:11.385048 8982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:11.369743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46434 (local address 127.31.235.131:36943)
0620 20:23:11.370381 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:11.370388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:11.370417 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:23:11.370478 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:11.370484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:11.370691 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:11.370860 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:11.372262 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:11.373291 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:11.374607 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:11.375163 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:11.375310 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:11.379720 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:11.379730 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:11.379745 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:11.379783 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:11.382769 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:11.383372 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:11.383378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:11.383381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:11.383488 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:11.383793 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:11.383796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:11.383797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:11.384115 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:11.384349 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:11.384547 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 20:23:11.384741 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":289,"threads_started":1}
I20260620 20:23:11.386466 8963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:23:11.468640 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 8840
I20260620 20:23:11.484762 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 8480
I20260620 20:23:11.486006 8542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:23:11.587370 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 8480
I20260620 20:23:11.607973 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:35007
--webserver_interface=127.31.235.190
--webserver_port=42579
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:35007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:23:11.937202 8985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:11.937503 8985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:11.937673 8985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:11.945134 8985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:23:11.945256 8985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:11.945330 8985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:11.945384 8985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:23:11.945441 8985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:23:11.957319 8985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:35007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:35007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=42579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:11.958680 8985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:11.959936 8985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:11.973311 8991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:11.973217 8990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:11.973587 8993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:11.974933 8985 server_base.cc:1034] running on GCE node
I20260620 20:23:11.975744 8985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:11.977385 8985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:11.978638 8985 hybrid_clock.cc:648] HybridClock initialized: now 1781986991978562 us; error 57 us; skew 500 ppm
I20260620 20:23:11.979070 8985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:23:11 dist-test-slave-zzwf krb5kdc[8467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986991, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:23:11.982259 8985 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:23:11.983830 8985 webserver.cc:460] Webserver started at http://127.31.235.190:42579/ using document root <none> and password file <none>
I20260620 20:23:11.984474 8985 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:11.984593 8985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:11.990198 8985 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 20:23:11.993378 9000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:11.994851 8985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:23:11.995050 8985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "015a8f0a189e40afabd09bf719a17d2b"
format_stamp: "Formatted at 2026-06-20 20:23:09 on dist-test-slave-zzwf"
server_key: "b4a51e17872373c403b5dc147631aed8"
server_key_iv: "c4167f992629b2ef357c4a54608beb49"
server_key_version: "encryptionkey@0"
I20260620 20:23:11.996064 8985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:12.022320 8985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:12.026849 8985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:12.027243 8985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:12.047981 8985 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:35007
I20260620 20:23:12.047988 9051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:35007 every 8 connection(s)
I20260620 20:23:12.049613 8985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:23:12.050366 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 8985
I20260620 20:23:12.051352 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:40821
--local_ip_for_outbound_sockets=127.31.235.129
--tserver_master_addrs=127.31.235.190:35007
--webserver_port=37421
--webserver_interface=127.31.235.129
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.235.129
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:23:12.057168 9052 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:23:12.060668 9052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b: Bootstrap starting.
I20260620 20:23:12.081009 9052 log.cc:824] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:23:12.089928 9052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:23:12.090492 9052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b: Bootstrap complete.
I20260620 20:23:12.100481 9052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015a8f0a189e40afabd09bf719a17d2b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35007 } }
I20260620 20:23:12.101533 9052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 015a8f0a189e40afabd09bf719a17d2b, State: Initialized, Role: FOLLOWER
I20260620 20:23:12.102241 9052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015a8f0a189e40afabd09bf719a17d2b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35007 } }
I20260620 20:23:12.102517 9052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:23:12.102650 9052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:23:12.102846 9052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:23:12.106797 9052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015a8f0a189e40afabd09bf719a17d2b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35007 } }
I20260620 20:23:12.107298 9052 leader_election.cc:302] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 015a8f0a189e40afabd09bf719a17d2b; no voters:
I20260620 20:23:12.107980 9052 leader_election.cc:288] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 20:23:12.108130 9056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:23:12.109405 9056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [term 2 LEADER]: Becoming Leader. State: Replica: 015a8f0a189e40afabd09bf719a17d2b, State: Running, Role: LEADER
I20260620 20:23:12.110018 9056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015a8f0a189e40afabd09bf719a17d2b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35007 } }
I20260620 20:23:12.110407 9052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:23:12.112939 9058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 015a8f0a189e40afabd09bf719a17d2b. Latest consensus state: current_term: 2 leader_uuid: "015a8f0a189e40afabd09bf719a17d2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015a8f0a189e40afabd09bf719a17d2b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35007 } } }
I20260620 20:23:12.112968 9057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "015a8f0a189e40afabd09bf719a17d2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "015a8f0a189e40afabd09bf719a17d2b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35007 } } }
I20260620 20:23:12.113893 9058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [sys.catalog]: This master's current role is: LEADER
I20260620 20:23:12.115599 9057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b [sys.catalog]: This master's current role is: LEADER
I20260620 20:23:12.122496 9062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:23:12.130035 9062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:23:12.131992 9062 catalog_manager.cc:1259] Loaded cluster ID: 0c7a9df4587540389fe02d549f17530d
I20260620 20:23:12.132136 9062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:23:12.138060 9062 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:23:12.140178 9062 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 015a8f0a189e40afabd09bf719a17d2b: Loaded TSK: 0
I20260620 20:23:12.142294 9062 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:23:12.400079 9054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:12.400367 9054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:12.400558 9054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:12.407866 9054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:12.408022 9054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:12.408191 9054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:23:12.417974 9054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.129.keytab
--principal=oryx/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:40821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=37421
--tserver_master_addrs=127.31.235.190:35007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:12.419255 9054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:12.420581 9054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:12.434918 9078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:12.435039 9081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:12.435688 9079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:12.436124 9054 server_base.cc:1034] running on GCE node
I20260620 20:23:12.436690 9054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:12.437703 9054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:12.438915 9054 hybrid_clock.cc:648] HybridClock initialized: now 1781986992438867 us; error 43 us; skew 500 ppm
I20260620 20:23:12.439210 9054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:23:12 dist-test-slave-zzwf krb5kdc[8467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986992, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:23:12.442312 9054 init.cc:377] Logged in from keytab as oryx/127.31.235.129@KRBTEST.COM (short username oryx)
I20260620 20:23:12.443800 9054 webserver.cc:460] Webserver started at http://127.31.235.129:37421/ using document root <none> and password file <none>
I20260620 20:23:12.444445 9054 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:12.444590 9054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:12.450119 9054 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:23:12.452795 9088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:12.454186 9054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:23:12.454345 9054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "b8df0be8c2864b109103c92ce553c565"
format_stamp: "Formatted at 2026-06-20 20:23:09 on dist-test-slave-zzwf"
server_key: "e4096ad409981668c992b8d5eb4dc578"
server_key_iv: "c9504bcf185416289930b878f95d5b41"
server_key_version: "encryptionkey@0"
I20260620 20:23:12.455260 9054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:12.485594 9054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:12.489967 9054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:12.490242 9054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:12.491192 9054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:23:12.492476 9054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:23:12.492563 9054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:12.492703 9054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:23:12.492799 9054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:12.525985 9054 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:40821
I20260620 20:23:12.526080 9200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:40821 every 8 connection(s)
I20260620 20:23:12.527423 9054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:23:12.533414 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 9054
I20260620 20:23:12.534157 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:34993
--local_ip_for_outbound_sockets=127.31.235.130
--tserver_master_addrs=127.31.235.190:35007
--webserver_port=37615
--webserver_interface=127.31.235.130
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.235.130
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:23:12 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:12 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:12.546468 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:12.530337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.129:43667)
0620 20:23:12.531100 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:12.531143 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:12.532756 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:12.534170 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:12.534183 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:12.534932 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:12.535712 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:12.535740 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:12.537260 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:12.537268 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:12.539406 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:12.539424 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:12.540436 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:12.540446 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:12.540719 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:12.541404 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:12.541426 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:12.545138 (+ 3712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":553,"threads_started":1}
W20260620 20:23:12.546495 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:12.530344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:43667 (local address 127.31.235.190:35007)
0620 20:23:12.531045 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:12.531060 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:12.531264 (+ 204us) server_negotiation.cc:407] Connection header received
0620 20:23:12.533264 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:12.533276 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:12.533664 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:12.533965 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:12.535917 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:12.537102 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:12.539564 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:12.540196 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:12.540330 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:12.545536 (+ 5206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.129:43667: BlockingRecv error: recv got EOF from 127.31.235.129:43667 (error 108)
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":277,"threads_started":1}
W20260620 20:23:12.547382 9201 heartbeater.cc:644] Failed to heartbeat to 127.31.235.190:35007 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.235.190:35007: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Jun 20 20:23:12 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:12 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:23:12.557969 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:12.548049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:50119 (local address 127.31.235.190:35007)
0620 20:23:12.548180 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:12.548184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:12.548200 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:23:12.548354 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:12.548358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:12.548411 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:12.548485 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:12.549356 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:12.550425 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:12.552469 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:12.552849 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:12.552941 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:12.557802 (+ 4861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.129:50119: BlockingRecv error: recv got EOF from 127.31.235.129:50119 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 20:23:12.558005 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:12.547914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.129:50119)
0620 20:23:12.548122 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:12.548142 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:12.548244 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:12.548541 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:12.548548 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:12.549012 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:12.549223 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:12.549237 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:12.550575 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:12.550583 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:12.552319 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:12.552336 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:12.553000 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:12.553009 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:12.553231 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:12.553899 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:12.553928 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:12.557644 (+ 3716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
Jun 20 20:23:12 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:12 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:12.567101 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:12.558799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.129:36725)
0620 20:23:12.558998 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:12.559016 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:12.559114 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:12.559597 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:12.559600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:12.559940 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:12.560082 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:12.560090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:12.561458 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:12.561462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:12.562425 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:12.562436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:12.563332 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:12.563337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:12.563467 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:12.564062 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:12.564079 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:12.566773 (+ 2694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 20:23:12.567327 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:12.559056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:36725 (local address 127.31.235.190:35007)
0620 20:23:12.559257 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:12.559266 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:12.559300 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:23:12.559356 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:12.559364 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:12.559458 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:12.559566 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:12.560201 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:12.561302 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:12.562547 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:12.563180 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:12.563299 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:12.567064 (+ 3765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.129:36725: BlockingRecv error: recv got EOF from 127.31.235.129:36725 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 20:23:12.567564 9201 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:23:12.857931 9205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:12.858238 9205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:12.858410 9205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:12.865556 9205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:12.865737 9205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:12.865926 9205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:23:12.875895 9205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.130.keytab
--principal=oryx/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:34993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=37615
--tserver_master_addrs=127.31.235.190:35007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:12.877256 9205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:12.878541 9205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:12.892609 9210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:12.892656 9213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:12.892527 9211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:12.893590 9205 server_base.cc:1034] running on GCE node
I20260620 20:23:12.894186 9205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:12.895128 9205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:12.896379 9205 hybrid_clock.cc:648] HybridClock initialized: now 1781986992896329 us; error 61 us; skew 500 ppm
I20260620 20:23:12.896713 9205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:23:12 dist-test-slave-zzwf krb5kdc[8467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986992, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:23:12.900125 9205 init.cc:377] Logged in from keytab as oryx/127.31.235.130@KRBTEST.COM (short username oryx)
I20260620 20:23:12.901468 9205 webserver.cc:460] Webserver started at http://127.31.235.130:37615/ using document root <none> and password file <none>
I20260620 20:23:12.902195 9205 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:12.902307 9205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:12.907452 9205 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:23:12.909929 9220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:12.911340 9205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:23:12.911538 9205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "a627dd6264334eb790f294b1c34de6c2"
format_stamp: "Formatted at 2026-06-20 20:23:10 on dist-test-slave-zzwf"
server_key: "f197cf766f85c527f9a2d8ee68d55782"
server_key_iv: "c31349c1ddaa046228d8a46e8591d3e2"
server_key_version: "encryptionkey@0"
I20260620 20:23:12.912524 9205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:12.934026 9205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:12.938391 9205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:12.938676 9205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:12.939658 9205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:23:12.941012 9205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:23:12.941094 9205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:12.941174 9205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:23:12.941277 9205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:12.974310 9205 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:34993
I20260620 20:23:12.974407 9332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:34993 every 8 connection(s)
I20260620 20:23:12.975713 9205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:23:12.978214 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 9205
I20260620 20:23:12.978816 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:36943
--local_ip_for_outbound_sockets=127.31.235.131
--tserver_master_addrs=127.31.235.190:35007
--webserver_port=45575
--webserver_interface=127.31.235.131
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:42919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.235.131
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:23:12 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:12 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:23:12.992449 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:12.978350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:55795 (local address 127.31.235.190:35007)
0620 20:23:12.978540 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:12.978547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:12.979885 (+ 1338us) server_negotiation.cc:407] Connection header received
0620 20:23:12.981761 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:12.981769 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:12.981862 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:12.981986 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:12.983364 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:12.984528 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:12.986609 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:12.987371 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:12.987514 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:12.992145 (+ 4631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.130:55795: BlockingRecv error: recv got EOF from 127.31.235.130:55795 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:23:12.992925 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:12.978819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.130:55795)
0620 20:23:12.979694 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:12.979749 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:12.981431 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:12.982178 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:12.982186 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:12.982659 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:12.983185 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:12.983201 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:12.984688 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:12.984696 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:12.986460 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:12.986472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:12.987592 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:12.987596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:12.987782 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:12.988464 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:12.988489 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:12.991906 (+ 3417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":371,"threads_started":1}
W20260620 20:23:12.993590 9333 heartbeater.cc:644] Failed to heartbeat to 127.31.235.190:35007 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.235.190:35007: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Jun 20 20:23:12 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:12 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:13.002087 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:12.994231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.130:34395)
0620 20:23:12.994384 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:12.994402 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:12.994523 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:12.994864 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:12.994868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:12.995187 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:12.995309 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:12.995316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:12.996271 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:12.996276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:12.997142 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:12.997152 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:12.997925 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:12.997930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:12.998098 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:12.998669 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:12.998686 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:13.001811 (+ 3125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 20:23:13.002396 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:12.994409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:34395 (local address 127.31.235.190:35007)
0620 20:23:12.994598 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:12.994605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:12.994623 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:23:12.994669 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:12.994673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:12.994735 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:12.994834 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:12.995442 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:12.996140 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:12.997280 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:12.997774 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:12.997917 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:13.002009 (+ 4092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.130:34395: BlockingRecv error: recv got EOF from 127.31.235.130:34395 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 20:23:13 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:13 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:13.010367 9335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:13.002763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.130:58021)
0620 20:23:13.002922 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:13.002943 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:13.003040 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:13.003434 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:13.003438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:13.003840 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:13.003997 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:13.004005 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.004936 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.004942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:13.005891 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:13.005902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.006545 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.006551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:13.006725 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:13.007280 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:13.007296 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:13.010078 (+ 2782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 20:23:13.010499 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:13.002855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:58021 (local address 127.31.235.190:35007)
0620 20:23:13.003167 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:13.003172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:13.003194 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:23:13.003253 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:13.003256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:13.003307 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:13.003411 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:13.004133 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.004810 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.006025 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.006410 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.006539 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:13.010261 (+ 3722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.130:58021: BlockingRecv error: recv got EOF from 127.31.235.130:58021 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":719,"threads_started":1}
W20260620 20:23:13.010891 9333 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:23:13.307890 9336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:13.308178 9336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:13.308269 9336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:13.315716 9336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:13.315830 9336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:13.315943 9336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:23:13.325955 9336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:42919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/krb5kdc/oryx_127.31.235.131.keytab
--principal=oryx/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:36943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=45575
--tserver_master_addrs=127.31.235.190:35007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:13.327196 9336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:13.328377 9336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:13.341046 9345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:13.341150 9343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:13.341212 9342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:13.341987 9336 server_base.cc:1034] running on GCE node
I20260620 20:23:13.342551 9336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:13.343447 9336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:13.344635 9336 hybrid_clock.cc:648] HybridClock initialized: now 1781986993344576 us; error 55 us; skew 500 ppm
I20260620 20:23:13.344944 9336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:23:13 dist-test-slave-zzwf krb5kdc[8467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781986993, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:23:13.348169 9336 init.cc:377] Logged in from keytab as oryx/127.31.235.131@KRBTEST.COM (short username oryx)
I20260620 20:23:13.349555 9336 webserver.cc:460] Webserver started at http://127.31.235.131:45575/ using document root <none> and password file <none>
I20260620 20:23:13.350242 9336 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:13.350392 9336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:13.355623 9336 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:23:13.358265 9352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:13.359619 9336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:23:13.359783 9336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "2ca1c598c07144c985728a9e320cfe88"
format_stamp: "Formatted at 2026-06-20 20:23:10 on dist-test-slave-zzwf"
server_key: "b3c0e8b6f7b8e482291ba75576305243"
server_key_iv: "0f9ecc2cfe4c638ea5be3b535a7f6b3e"
server_key_version: "encryptionkey@0"
I20260620 20:23:13.360738 9336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:13.384178 9336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:13.388170 9336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:13.388460 9336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:13.389441 9336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:23:13.390774 9336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:23:13.390913 9336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:13.391045 9336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:23:13.391131 9336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:13.423334 9336 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:36943
I20260620 20:23:13.423393 9464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:36943 every 8 connection(s)
I20260620 20:23:13.424738 9336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:23:13.431838 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 9336
Jun 20 20:23:13 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:13 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 20:23:13.440583 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:13.427331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:60921 (local address 127.31.235.190:35007)
0620 20:23:13.427515 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:13.427519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:13.428571 (+ 1052us) server_negotiation.cc:407] Connection header received
0620 20:23:13.430564 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:13.430571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:13.430675 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:13.430784 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:13.432551 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.433465 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.434981 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.435403 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.435547 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:13.440305 (+ 4758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.131:60921: BlockingRecv error: recv got EOF from 127.31.235.131:60921 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:23:13.441332 9467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:13.427769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.131:60921)
0620 20:23:13.428425 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:13.428460 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:13.430308 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:13.431009 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:13.431016 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:13.431588 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:13.432369 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:13.432397 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.433632 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.433637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:13.434830 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:13.434851 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.435627 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.435633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:13.435857 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:13.436546 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:13.436566 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:13.440076 (+ 3510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":255,"threads_started":1}
W20260620 20:23:13.442087 9465 heartbeater.cc:644] Failed to heartbeat to 127.31.235.190:35007 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.235.190:35007: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
I20260620 20:23:13.447463 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:13.432912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37490 (local address 127.31.235.190:35007)
0620 20:23:13.433079 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:13.433085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:13.433468 (+ 383us) server_negotiation.cc:407] Connection header received
0620 20:23:13.433657 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:13.433664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:13.433737 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:13.433843 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:13.435278 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.436069 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.437315 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.437800 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.437924 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:13.440586 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:13.440597 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:13.440621 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:13.440665 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:13.444319 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:13.444953 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:13.444965 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:13.444975 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:13.445194 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:13.445634 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:13.445637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:13.445639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:13.446065 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:13.446276 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:13.446802 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 20:23:13.447157 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 20:23:13 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:13 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:13.451048 9467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:13.442768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.131:33393)
0620 20:23:13.442952 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:13.442972 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:13.443073 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:13.443381 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:13.443384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:13.443659 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:13.443802 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:13.443809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.445065 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.445070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:13.446217 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:13.446230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.447153 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.447157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:13.447276 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:13.447843 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:13.447860 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:13.450772 (+ 2912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 20:23:13.451273 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:13.442945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:33393 (local address 127.31.235.190:35007)
0620 20:23:13.443093 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:13.443099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:13.443117 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:23:13.443176 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:13.443182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:13.443253 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:13.443357 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:13.443934 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.444869 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.446350 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.447002 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.447125 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:13.450931 (+ 3806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.131:33393: BlockingRecv error: recv got EOF from 127.31.235.131:33393 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 20:23:13 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:13 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:13.459002 9467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:13.451694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.131:56445)
0620 20:23:13.451834 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:13.451849 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:13.451923 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:13.452420 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:13.452424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:13.452703 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:13.452849 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:13.452856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.453839 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.453846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:13.454737 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:13.454747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.455423 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.455428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:13.455597 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:13.456079 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:13.456095 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:13.458691 (+ 2596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 20:23:13.459306 9465 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:23:13.459151 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:13.451952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:56445 (local address 127.31.235.190:35007)
0620 20:23:13.452145 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:13.452149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:13.452161 (+ 12us) server_negotiation.cc:407] Connection header received
0620 20:23:13.452195 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:13.452198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:13.452291 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:13.452397 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:13.452979 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.453712 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.454873 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.455286 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.455414 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:13.458904 (+ 3490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.131:56445: BlockingRecv error: recv got EOF from 127.31.235.131:56445 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Jun 20 20:23:13 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:13 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:13.577363 9469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:13.568222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.129:34833)
0620 20:23:13.568746 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:13.568763 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:13.568899 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:13.569260 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:13.569265 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:13.569759 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:13.569955 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:13.569965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.571156 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.571163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:13.572612 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:13.572629 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.573443 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.573448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:13.573672 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:13.574249 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:13.574267 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:13.577104 (+ 2837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":311,"threads_started":1}
W20260620 20:23:13.577499 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:13.568347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:34833 (local address 127.31.235.190:35007)
0620 20:23:13.568635 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:13.568641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:13.568870 (+ 229us) server_negotiation.cc:407] Connection header received
0620 20:23:13.569048 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:13.569052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:13.569115 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:13.569195 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:13.570074 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.570934 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.572786 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:13.573284 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:13.573395 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:13.577300 (+ 3905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.129:34833: BlockingRecv error: recv got EOF from 127.31.235.129:34833 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
Jun 20 20:23:14 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:14 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:14.020670 9470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:14.011576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.130:46669)
0620 20:23:14.012114 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:14.012133 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:14.012321 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:14.012674 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:14.012690 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:14.013145 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:14.013327 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:14.013336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:14.014620 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:14.014625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:14.015850 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:14.015863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:14.016717 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:14.016723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:14.016847 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:14.017454 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:14.017478 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:14.020443 (+ 2965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":295,"threads_started":1}
W20260620 20:23:14.020864 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:14.011680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:46669 (local address 127.31.235.190:35007)
0620 20:23:14.011952 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:14.011960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:14.012245 (+ 285us) server_negotiation.cc:407] Connection header received
0620 20:23:14.012450 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:14.012457 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:14.012523 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:14.012628 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:14.013477 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:14.014458 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:14.015990 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:14.016532 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:14.016628 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:14.020636 (+ 4008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.130:46669: BlockingRecv error: recv got EOF from 127.31.235.130:46669 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 20 20:23:14 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:14 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:14.468452 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:14.459932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.131:33185)
0620 20:23:14.460409 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:14.460425 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:14.460541 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:14.460836 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:14.460840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:14.461281 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:14.461461 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:14.461469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:14.462454 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:14.462458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:14.463387 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:14.463397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:14.464045 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:14.464050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:14.464163 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:14.464854 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:14.464872 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:14.468126 (+ 3254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":272,"threads_started":1}
W20260620 20:23:14.468541 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:14.459979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:33185 (local address 127.31.235.190:35007)
0620 20:23:14.460202 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:14.460212 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:14.460509 (+ 297us) server_negotiation.cc:407] Connection header received
0620 20:23:14.460638 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:14.460642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:14.460703 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:14.460780 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:14.461643 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:14.462331 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:14.463526 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:14.463895 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:14.463988 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:14.468322 (+ 4334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.131:33185: BlockingRecv error: recv got EOF from 127.31.235.131:33185 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 20:23:14 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:14 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:14.586568 9472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:14.578387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.129:53327)
0620 20:23:14.578885 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:14.578901 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:14.579058 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:14.579380 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:14.579385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:14.579806 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:14.579976 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:14.579985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:14.580978 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:14.580986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:14.581895 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:14.581906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:14.582606 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:14.582612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:14.582775 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:14.583391 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:14.583407 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:14.586227 (+ 2820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":264,"threads_started":1}
W20260620 20:23:14.586808 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:14.578436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:53327 (local address 127.31.235.190:35007)
0620 20:23:14.578672 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:14.578679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:14.578991 (+ 312us) server_negotiation.cc:407] Connection header received
0620 20:23:14.579178 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:14.579185 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:14.579249 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:14.579357 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:14.580117 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:14.580852 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:14.582034 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:14.582449 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:14.582592 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:14.586422 (+ 3830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.129:53327: BlockingRecv error: recv got EOF from 127.31.235.129:53327 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 20:23:15 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:15 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:15.029284 9473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:15.021644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.130:54881)
0620 20:23:15.022121 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:15.022142 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:15.022256 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:15.022543 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:15.022546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:15.022937 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:15.023100 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:15.023107 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:15.024045 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:15.024049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:15.024945 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:15.024955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:15.025588 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:15.025592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:15.025696 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:15.026357 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:15.026374 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:15.029049 (+ 2675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":259,"threads_started":1}
W20260620 20:23:15.029469 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:15.021689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:54881 (local address 127.31.235.190:35007)
0620 20:23:15.021917 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:15.021926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:15.022223 (+ 297us) server_negotiation.cc:407] Connection header received
0620 20:23:15.022369 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:15.022372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:15.022430 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:15.022501 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:15.023216 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:15.023939 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:15.025060 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:15.025430 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:15.025592 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:15.029223 (+ 3631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.130:54881: BlockingRecv error: recv got EOF from 127.31.235.130:54881 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 20:23:15 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:15 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:15.477820 9474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:15.469327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.131:39755)
0620 20:23:15.469902 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:15.469919 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:15.470073 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:15.470423 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:15.470428 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:15.470873 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:15.471059 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:15.471078 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:15.472150 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:15.472155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:15.473045 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:15.473056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:15.473858 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:15.473864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:15.474049 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:15.474589 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:15.474607 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:15.477463 (+ 2856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":337,"threads_started":1}
W20260620 20:23:15.478017 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:15.469586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:39755 (local address 127.31.235.190:35007)
0620 20:23:15.469802 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:15.469808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:15.469998 (+ 190us) server_negotiation.cc:407] Connection header received
0620 20:23:15.470208 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:15.470220 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:15.470292 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:15.470394 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:15.471253 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:15.471988 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:15.473221 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:15.473711 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:15.473847 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:15.477718 (+ 3871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.131:39755: BlockingRecv error: recv got EOF from 127.31.235.131:39755 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 20:23:15 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:15 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:15.595381 9475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:15.587489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.129:50381)
0620 20:23:15.587993 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:15.588009 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:15.588129 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:15.588458 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:15.588461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:15.588781 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:15.588944 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:15.588953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:15.589966 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:15.589970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:15.590871 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:15.590883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:15.591500 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:15.591505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:15.591642 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:15.592317 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:15.592335 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:15.595094 (+ 2759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":273,"threads_started":1}
W20260620 20:23:15.595463 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:15.587555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:50381 (local address 127.31.235.190:35007)
0620 20:23:15.587716 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:15.587720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:15.588082 (+ 362us) server_negotiation.cc:407] Connection header received
0620 20:23:15.588220 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:15.588223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:15.588281 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:15.588357 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:15.589061 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:15.589819 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:15.590991 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:15.591360 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:15.591447 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:15.595278 (+ 3831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.129:50381: BlockingRecv error: recv got EOF from 127.31.235.129:50381 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 20:23:16 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:16 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:16.038913 9476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:16.030150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.130:60189)
0620 20:23:16.030716 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:16.030740 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:16.030909 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:16.031274 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:16.031279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:16.031658 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:16.031836 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:16.031846 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:16.032914 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:16.032922 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:16.033937 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:16.033950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:16.034736 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:16.034742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:16.034942 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:16.035611 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:16.035630 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:16.038513 (+ 2883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":315,"threads_started":1}
W20260620 20:23:16.038985 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:16.030333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:60189 (local address 127.31.235.190:35007)
0620 20:23:16.030576 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:16.030584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:16.030831 (+ 247us) server_negotiation.cc:407] Connection header received
0620 20:23:16.031044 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:16.031051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:16.031126 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:16.031245 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:16.031983 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:16.032775 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:16.034085 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:16.034543 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:16.034717 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:16.038727 (+ 4010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.130:60189: BlockingRecv error: recv got EOF from 127.31.235.130:60189 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 20 20:23:16 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:16 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:16.486992 9477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:16.479010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.131:52695)
0620 20:23:16.479543 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:16.479560 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:16.479670 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:16.479965 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:16.479968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:16.480285 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:16.480461 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:16.480469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:16.481439 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:16.481443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:16.482400 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:16.482411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:16.483026 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:16.483030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:16.483151 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:16.483735 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:16.483751 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:16.486741 (+ 2990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":272,"threads_started":1}
W20260620 20:23:16.487123 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:16.479069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:52695 (local address 127.31.235.190:35007)
0620 20:23:16.479334 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:16.479341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:16.479642 (+ 301us) server_negotiation.cc:407] Connection header received
0620 20:23:16.479784 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:16.479788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:16.479849 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:16.479923 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:16.480586 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:16.481311 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:16.482527 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:16.482911 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:16.483003 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:16.486929 (+ 3926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.131:52695: BlockingRecv error: recv got EOF from 127.31.235.131:52695 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 20:23:16 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:16 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:16.604481 9478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:16.596280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.129:40761)
0620 20:23:16.596739 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:16.596754 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:16.596875 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:16.597133 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:16.597136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:16.597505 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:16.597653 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:16.597661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:16.598713 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:16.598717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:16.599833 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:16.599845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:16.600641 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:16.600646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:16.600820 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:16.601395 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:16.601412 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:16.604189 (+ 2777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":262,"threads_started":1}
W20260620 20:23:16.604744 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:16.596357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:40761 (local address 127.31.235.190:35007)
0620 20:23:16.596591 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:16.596598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:16.596833 (+ 235us) server_negotiation.cc:407] Connection header received
0620 20:23:16.596955 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:16.596958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:16.597016 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:16.597092 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:16.597799 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:16.598562 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:16.599978 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:16.600476 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:16.600589 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:16.604460 (+ 3871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.129:40761: BlockingRecv error: recv got EOF from 127.31.235.129:40761 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 20:23:17 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:17 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:17.048254 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:17.039841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.130:51415)
0620 20:23:17.040388 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:17.040409 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:17.040571 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:17.040922 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:17.040927 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:17.041327 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:17.041470 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:17.041478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:17.042496 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:17.042502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:17.043456 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:17.043467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:17.044192 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:17.044198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:17.044378 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:17.044975 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:17.045040 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:17.048024 (+ 2984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":276,"threads_started":1}
W20260620 20:23:17.048543 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:17.039935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:51415 (local address 127.31.235.190:35007)
0620 20:23:17.040199 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:17.040205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:17.040502 (+ 297us) server_negotiation.cc:407] Connection header received
0620 20:23:17.040718 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:17.040722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:17.040789 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:17.040898 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:17.041636 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:17.042345 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:17.043609 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:17.044038 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:17.044169 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:17.048288 (+ 4119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.130:51415: BlockingRecv error: recv got EOF from 127.31.235.130:51415 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Jun 20 20:23:17 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:17 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:17.500960 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:17.488040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.131:59369)
0620 20:23:17.488588 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:17.488604 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:17.488715 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:17.489061 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:17.489065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:17.489533 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:17.489693 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:17.489702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:17.490729 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:17.490733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:17.491812 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:17.491823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:17.492488 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:17.492493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:17.492619 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:17.497779 (+ 5160us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:17.497796 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:17.500588 (+ 2792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":328,"threads_started":1}
W20260620 20:23:17.501006 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:17.488085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:59369 (local address 127.31.235.190:35007)
0620 20:23:17.488270 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:17.488275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:17.488681 (+ 406us) server_negotiation.cc:407] Connection header received
0620 20:23:17.488814 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:17.488817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:17.488882 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:17.488962 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:17.489848 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:17.490589 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:17.491953 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:17.492347 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:17.492439 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:17.500773 (+ 8334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.131:59369: BlockingRecv error: recv got EOF from 127.31.235.131:59369 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 20:23:17 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:17 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:17.613657 9481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:17.605465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.129:37531)
0620 20:23:17.606013 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:17.606029 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:17.606136 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:17.606458 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:17.606461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:17.606845 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:17.606994 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:17.607002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:17.608052 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:17.608056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:17.608994 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:17.609007 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:17.609679 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:17.609683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:17.609804 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:17.610434 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:17.610451 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:17.613347 (+ 2896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":277,"threads_started":1}
W20260620 20:23:17.613912 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:17.605630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:37531 (local address 127.31.235.190:35007)
0620 20:23:17.605819 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:17.605824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:17.606133 (+ 309us) server_negotiation.cc:407] Connection header received
0620 20:23:17.606261 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:17.606265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:17.606330 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:17.606413 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:17.607147 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:17.607891 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:17.609130 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:17.609562 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:17.609668 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:17.613627 (+ 3959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.129:37531: BlockingRecv error: recv got EOF from 127.31.235.129:37531 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 20:23:18 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:18 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:18.058440 9482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:18.049330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.130:59739)
0620 20:23:18.049940 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:18.049963 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:18.050116 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:18.050805 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:18.050810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:18.051361 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:18.051582 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:18.051597 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:18.052712 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:18.052720 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:18.053780 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:18.053792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:18.054542 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:18.054548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:18.054725 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:18.055380 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:18.055398 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:18.058185 (+ 2787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":308,"threads_started":1}
W20260620 20:23:18.058669 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:18.049535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:59739 (local address 127.31.235.190:35007)
0620 20:23:18.049733 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:18.049739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:18.050051 (+ 312us) server_negotiation.cc:407] Connection header received
0620 20:23:18.050501 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:18.050505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:18.050643 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:18.050775 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:18.051734 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:18.052563 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:18.053948 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:18.054385 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:18.054519 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:18.058402 (+ 3883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.130:59739: BlockingRecv error: recv got EOF from 127.31.235.130:59739 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 20:23:18 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:18 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:18.510049 9483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:18.502045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.131:34295)
0620 20:23:18.502746 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:18.502762 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:18.502902 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:18.503352 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:18.503357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:18.503826 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:18.503995 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:18.504004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:18.504930 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:18.504934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:18.505829 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:18.505839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:18.506491 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:18.506495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:18.506609 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:18.507157 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:18.507173 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:18.509797 (+ 2624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":437,"threads_started":1}
W20260620 20:23:18.510254 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:18.502174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:34295 (local address 127.31.235.190:35007)
0620 20:23:18.502398 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:18.502404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:18.502852 (+ 448us) server_negotiation.cc:407] Connection header received
0620 20:23:18.503101 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:18.503107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:18.503190 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:18.503293 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:18.504127 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:18.504811 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:18.505948 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:18.506364 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:18.506457 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:18.509992 (+ 3535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.131:34295: BlockingRecv error: recv got EOF from 127.31.235.131:34295 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 20:23:18 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:18 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:18.622723 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:18.614545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.129:40081)
0620 20:23:18.615101 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:18.615117 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:18.615226 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:18.615505 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:18.615508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:18.615870 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:18.616084 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:18.616093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:18.617196 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:18.617201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:18.618143 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:18.618154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:18.618828 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:18.618834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:18.619004 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:18.619570 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:18.619586 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:18.622396 (+ 2810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":325,"threads_started":1}
W20260620 20:23:18.622822 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:18.614653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:40081 (local address 127.31.235.190:35007)
0620 20:23:18.614819 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:18.614823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:18.615211 (+ 388us) server_negotiation.cc:407] Connection header received
0620 20:23:18.615326 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:18.615330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:18.615391 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:18.615470 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:18.616284 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:18.617054 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:18.618287 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:18.618680 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:18.618812 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:18.622590 (+ 3778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.129:40081: BlockingRecv error: recv got EOF from 127.31.235.129:40081 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 20:23:19 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:19 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:19.067469 9485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:19.059269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.130:59413)
0620 20:23:19.059859 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:19.059880 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:19.059993 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:19.060276 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:19.060279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:19.060626 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:19.060774 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:19.060782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:19.061729 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:19.061733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:19.062633 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:19.062644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:19.063276 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:19.063281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:19.063413 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:19.064019 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:19.064036 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:19.067140 (+ 3104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":389,"threads_started":1}
W20260620 20:23:19.067682 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:19.059315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:59413 (local address 127.31.235.190:35007)
0620 20:23:19.059561 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:19.059571 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:19.059970 (+ 399us) server_negotiation.cc:407] Connection header received
0620 20:23:19.060094 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:19.060097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:19.060157 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:19.060238 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:19.060894 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:19.061611 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:19.062761 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:19.063150 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:19.063241 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:19.067395 (+ 4154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.130:59413: BlockingRecv error: recv got EOF from 127.31.235.130:59413 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 20:23:19 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:19 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:19.519760 9486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:19.511012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.131:58855)
0620 20:23:19.511606 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:19.511622 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:19.511810 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:19.512296 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:19.512300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:19.512639 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:19.512817 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:19.512825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:19.513862 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:19.513866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:19.514856 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:19.514870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:19.515930 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:19.515937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:19.516066 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:19.516817 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:19.516833 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:19.519549 (+ 2716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":329,"threads_started":1}
W20260620 20:23:19.520056 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:19.511103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:58855 (local address 127.31.235.190:35007)
0620 20:23:19.511286 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:19.511291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:19.511705 (+ 414us) server_negotiation.cc:407] Connection header received
0620 20:23:19.512003 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:19.512007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:19.512073 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:19.512164 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:19.512970 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:19.513715 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:19.515027 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:19.515766 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:19.515877 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:19.519793 (+ 3916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.131:58855: BlockingRecv error: recv got EOF from 127.31.235.131:58855 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 20:23:19 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:19 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:19.632392 9487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:19.623747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.129:38241)
0620 20:23:19.624403 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:19.624418 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:19.624578 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:19.624919 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:19.624924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:19.625552 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:19.625816 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:19.625828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:19.626934 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:19.626941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:19.627848 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:19.627858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:19.628564 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:19.628575 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:19.628738 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:19.629364 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:19.629380 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:19.632145 (+ 2765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":283,"threads_started":1}
W20260620 20:23:19.632555 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:19.623909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:38241 (local address 127.31.235.190:35007)
0620 20:23:19.624109 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:19.624118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:19.624513 (+ 395us) server_negotiation.cc:407] Connection header received
0620 20:23:19.624702 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:19.624707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:19.624784 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:19.624887 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:19.625980 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:19.626792 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:19.627987 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:19.628415 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:19.628551 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:19.632347 (+ 3796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.129:38241: BlockingRecv error: recv got EOF from 127.31.235.129:38241 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 20:23:20 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:20 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:20.077293 9488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:20.068507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.130:50113)
0620 20:23:20.069115 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:20.069135 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:20.069251 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:20.069696 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:20.069700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:20.070139 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:20.070325 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:20.070335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:20.071365 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:20.071369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:20.072281 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:20.072291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:20.072987 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:20.072993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:20.073173 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:20.073959 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:20.073983 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:20.076928 (+ 2945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":294,"threads_started":1}
W20260620 20:23:20.077363 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:20.068613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:50113 (local address 127.31.235.190:35007)
0620 20:23:20.068818 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:20.068823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:20.069241 (+ 418us) server_negotiation.cc:407] Connection header received
0620 20:23:20.069387 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:20.069392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:20.069459 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:20.069628 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:20.070481 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:20.071235 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:20.072415 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:20.072820 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:20.072921 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:20.077115 (+ 4194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.130:50113: BlockingRecv error: recv got EOF from 127.31.235.130:50113 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Jun 20 20:23:20 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:20 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:20.529446 9489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:20.520754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.131:53529)
0620 20:23:20.521398 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:20.521415 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:20.521554 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:20.521912 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:20.521915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:20.522304 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:20.522467 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:20.522476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:20.523565 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:20.523569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:20.524451 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:20.524461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:20.525337 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:20.525343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:20.525516 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:20.526136 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:20.526153 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:20.529192 (+ 3039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":282,"threads_started":1}
W20260620 20:23:20.529579 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:20.520899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:53529 (local address 127.31.235.190:35007)
0620 20:23:20.521083 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:20.521087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:20.521531 (+ 444us) server_negotiation.cc:407] Connection header received
0620 20:23:20.521679 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:20.521683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:20.521762 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:20.521860 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:20.522618 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:20.523447 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:20.524553 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:20.525172 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:20.525288 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:20.529364 (+ 4076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.131:53529: BlockingRecv error: recv got EOF from 127.31.235.131:53529 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 20:23:20 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:20 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:20.641669 9490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:20.633299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.129:48637)
0620 20:23:20.633904 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:20.633919 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:20.634092 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:20.634505 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:20.634510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:20.635026 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:20.635221 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:20.635233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:20.636327 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:20.636331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:20.637222 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:20.637232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:20.637969 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:20.637973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:20.638089 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:20.638649 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:20.638665 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:20.641363 (+ 2698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":351,"threads_started":1}
W20260620 20:23:20.642007 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:20.633559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:48637 (local address 127.31.235.190:35007)
0620 20:23:20.633780 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:20.633787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:20.634019 (+ 232us) server_negotiation.cc:407] Connection header received
0620 20:23:20.634223 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:20.634229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:20.634328 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:20.634429 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:20.635393 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:20.636197 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:20.637346 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:20.637815 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:20.637904 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:20.641626 (+ 3722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.129:48637: BlockingRecv error: recv got EOF from 127.31.235.129:48637 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 20:23:21 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:21 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:21.086850 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:21.078360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.130:58833)
0620 20:23:21.078891 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:21.078912 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:21.079028 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:21.079353 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:21.079356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:21.079710 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:21.079875 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:21.079884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:21.081117 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:21.081122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:21.082122 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:21.082172 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:21.082957 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:21.082961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:21.083092 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:21.083775 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:21.083793 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:21.086612 (+ 2819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":291,"threads_started":1}
W20260620 20:23:21.087054 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:21.078499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:58833 (local address 127.31.235.190:35007)
0620 20:23:21.078759 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:21.078766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:21.078997 (+ 231us) server_negotiation.cc:407] Connection header received
0620 20:23:21.079130 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:21.079134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:21.079195 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:21.079310 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:21.080030 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:21.080941 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:21.082344 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:21.082769 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:21.082875 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:21.086815 (+ 3940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.130:58833: BlockingRecv error: recv got EOF from 127.31.235.130:58833 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Jun 20 20:23:21 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:21 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:21.538821 9492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:21.530318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.131:48813)
0620 20:23:21.530850 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:21.530866 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:21.530978 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:21.531387 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:21.531391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:21.531743 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:21.531888 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:21.531896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:21.532909 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:21.532914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:21.533985 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:21.533997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:21.534850 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:21.534858 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:21.535008 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:21.535717 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:21.535734 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:21.538570 (+ 2836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":284,"threads_started":1}
W20260620 20:23:21.539074 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:21.530477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:48813 (local address 127.31.235.190:35007)
0620 20:23:21.530672 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:21.530677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:21.530956 (+ 279us) server_negotiation.cc:407] Connection header received
0620 20:23:21.531129 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:21.531134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:21.531201 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:21.531337 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:21.532044 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:21.532757 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:21.534106 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:21.534672 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:21.534793 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:21.538777 (+ 3984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.131:48813: BlockingRecv error: recv got EOF from 127.31.235.131:48813 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 20:23:21 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:21 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:21.651046 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:21.642583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.129:49401)
0620 20:23:21.643046 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:21.643089 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:21.643218 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:21.643564 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:21.643569 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:21.643964 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:21.644127 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:21.644135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:21.645184 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:21.645191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:21.646184 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:21.646195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:21.646995 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:21.647002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:21.647173 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:21.647778 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:21.647795 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:21.650814 (+ 3019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":228,"threads_started":1}
W20260620 20:23:21.651208 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:21.643025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:49401 (local address 127.31.235.190:35007)
0620 20:23:21.643230 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:21.643236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:21.643256 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:23:21.643323 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:21.643328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:21.643393 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:21.643516 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:21.644273 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:21.645024 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:21.646351 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:21.646835 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:21.646964 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:21.650987 (+ 4023us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.129:49401: BlockingRecv error: recv got EOF from 127.31.235.129:49401 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 20:23:22 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:22 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:22.096431 9494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:22.087862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.130:33107)
0620 20:23:22.088514 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:22.088543 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:22.088688 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:22.089036 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:22.089041 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:22.089458 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:22.089698 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:22.089708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:22.090724 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:22.090731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:22.091704 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:22.091714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:22.092428 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:22.092434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:22.092609 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:22.093242 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:22.093259 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:22.096183 (+ 2924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":298,"threads_started":1}
W20260620 20:23:22.096614 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:22.087945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:33107 (local address 127.31.235.190:35007)
0620 20:23:22.088142 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:22.088151 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:22.088618 (+ 467us) server_negotiation.cc:407] Connection header received
0620 20:23:22.088820 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:22.088825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:22.088897 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:22.089003 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:22.089842 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:22.090589 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:22.091848 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:22.092268 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:22.092408 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:22.096390 (+ 3982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.130:33107: BlockingRecv error: recv got EOF from 127.31.235.130:33107 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 20:23:22 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:22 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:22.552203 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:22.539870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.131:60809)
0620 20:23:22.540382 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:22.540398 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:22.540530 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:22.540822 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:22.540825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:22.541176 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:22.541334 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:22.541343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:22.542383 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:22.542386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:22.543296 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:22.543309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:22.544030 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:22.544035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:22.544150 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:22.548787 (+ 4637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:22.548804 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:22.551837 (+ 3033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":306,"threads_started":1}
W20260620 20:23:22.552304 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:22.539917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:60809 (local address 127.31.235.190:35007)
0620 20:23:22.540145 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:22.540156 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:22.540498 (+ 342us) server_negotiation.cc:407] Connection header received
0620 20:23:22.540630 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:22.540634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:22.540698 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:22.540785 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:22.541468 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:22.542257 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:22.543424 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:22.543888 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:22.543993 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:22.552045 (+ 8052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.131:60809: BlockingRecv error: recv got EOF from 127.31.235.131:60809 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 20:23:22 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:22 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:22.660625 9496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:22.652009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.129:54741)
0620 20:23:22.652521 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:22.652537 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:22.652695 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:22.653031 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:22.653037 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:22.653481 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:22.653704 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:22.653715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:22.654948 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:22.654955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:22.655903 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:22.655914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:22.656642 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:22.656647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:22.656812 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:22.657375 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:22.657391 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:22.660276 (+ 2885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":293,"threads_started":1}
W20260620 20:23:22.660727 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:22.652081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:54741 (local address 127.31.235.190:35007)
0620 20:23:22.652411 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:22.652417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:22.652625 (+ 208us) server_negotiation.cc:407] Connection header received
0620 20:23:22.652821 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:22.652828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:22.652897 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:22.653001 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:22.653839 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:22.654823 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:22.656040 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:22.656498 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:22.656630 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:22.660470 (+ 3840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.129:54741: BlockingRecv error: recv got EOF from 127.31.235.129:54741 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Jun 20 20:23:23 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:23 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:23.107088 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:23.097628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.130:45999)
0620 20:23:23.098161 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:23.098182 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:23.098296 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:23.098866 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:23.098870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:23.099270 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:23.099454 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:23.099462 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:23.100756 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:23.100761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:23.102259 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:23.102271 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:23.102987 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:23.102991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:23.103097 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:23.103750 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:23.103768 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:23.106851 (+ 3083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":309,"threads_started":1}
W20260620 20:23:23.107340 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:23.097814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:45999 (local address 127.31.235.190:35007)
0620 20:23:23.097979 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:23.097983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:23.098287 (+ 304us) server_negotiation.cc:407] Connection header received
0620 20:23:23.098503 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:23.098584 (+ 81us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:23.098665 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:23.098786 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:23.099611 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:23.100573 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:23.102415 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:23.102842 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:23.102943 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:23.107072 (+ 4129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.130:45999: BlockingRecv error: recv got EOF from 127.31.235.130:45999 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 20:23:23 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:23 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:23.562506 9498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:23.553368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.131:53883)
0620 20:23:23.553983 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:23.554001 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:23.554118 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:23.554531 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:23.554536 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:23.554983 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:23.555131 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:23.555140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:23.556471 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:23.556480 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:23.557560 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:23.557579 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:23.558354 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:23.558360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:23.558546 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:23.559160 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:23.559178 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:23.562150 (+ 2972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":360,"threads_started":1}
W20260620 20:23:23.562661 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:23.553630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:53883 (local address 127.31.235.190:35007)
0620 20:23:23.553908 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:23.553914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:23.554090 (+ 176us) server_negotiation.cc:407] Connection header received
0620 20:23:23.554279 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:23.554283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:23.554348 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:23.554499 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:23.555307 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:23.556318 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:23.557722 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:23.558173 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:23.558343 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:23.562356 (+ 4013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.131:53883: BlockingRecv error: recv got EOF from 127.31.235.131:53883 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Jun 20 20:23:23 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:23 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:23.671823 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:23.661661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.129:56073)
0620 20:23:23.662280 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:23.662297 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:23.662415 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:23.662816 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:23.662819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:23.663216 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:23.663391 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:23.663401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:23.665070 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:23.665075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:23.666689 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:23.666707 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:23.667421 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:23.667426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:23.667604 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:23.668231 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:23.668248 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:23.671473 (+ 3225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":360,"threads_started":1}
W20260620 20:23:23.672057 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:23.661805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:56073 (local address 127.31.235.190:35007)
0620 20:23:23.662059 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:23.662069 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:23.662381 (+ 312us) server_negotiation.cc:407] Connection header received
0620 20:23:23.662555 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:23.662564 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:23.662663 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:23.662799 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:23.663536 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:23.664865 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:23.666813 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:23.667263 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:23.667412 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:23.671696 (+ 4284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.129:56073: BlockingRecv error: recv got EOF from 127.31.235.129:56073 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 20:23:24 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:24 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:24.116993 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:24.108090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.130:36111)
0620 20:23:24.108628 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:24.108648 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:24.108807 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:24.109154 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:24.109159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:24.109640 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:24.109823 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:24.109832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:24.110902 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:24.110910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:24.111806 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:24.111818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:24.112512 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:24.112518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:24.112702 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:24.113436 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:24.113459 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:24.116646 (+ 3187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":259,"threads_started":1}
W20260620 20:23:24.117097 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:24.108173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:36111 (local address 127.31.235.190:35007)
0620 20:23:24.108396 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:24.108405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:24.108732 (+ 327us) server_negotiation.cc:407] Connection header received
0620 20:23:24.108949 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:24.108959 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:24.109019 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:24.109122 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:24.109971 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:24.110767 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:24.111957 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:24.112362 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:24.112497 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:24.116831 (+ 4334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.130:36111: BlockingRecv error: recv got EOF from 127.31.235.130:36111 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 20:23:24 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:24 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:24.571911 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:24.563503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.131:34595)
0620 20:23:24.564188 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:24.564209 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:24.564378 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:24.564744 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:24.564756 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:24.565217 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:24.565395 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:24.565403 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:24.566412 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:24.566416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:24.567296 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:24.567307 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:24.568003 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:24.568008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:24.568138 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:24.568853 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:24.568872 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:24.571663 (+ 2791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":425,"threads_started":1}
W20260620 20:23:24.572140 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:24.563528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:34595 (local address 127.31.235.190:35007)
0620 20:23:24.563783 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:24.563792 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:24.564300 (+ 508us) server_negotiation.cc:407] Connection header received
0620 20:23:24.564521 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:24.564528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:24.564612 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:24.564691 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:24.565567 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:24.566282 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:24.567415 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:24.567815 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:24.567907 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:24.571881 (+ 3974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.131:34595: BlockingRecv error: recv got EOF from 127.31.235.131:34595 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 20:23:24 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:24 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:24.680595 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:24.672847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.129:51839)
0620 20:23:24.673387 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:24.673403 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:24.673546 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:24.673841 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:24.673844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:24.674230 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:24.674375 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:24.674382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:24.675342 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:24.675346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:24.676259 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:24.676270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:24.676969 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:24.676973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:24.677095 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:24.677738 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:24.677755 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:24.680376 (+ 2621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":320,"threads_started":1}
W20260620 20:23:24.680895 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:24.672879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:51839 (local address 127.31.235.190:35007)
0620 20:23:24.673068 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:24.673072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:24.673521 (+ 449us) server_negotiation.cc:407] Connection header received
0620 20:23:24.673655 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:24.673658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:24.673716 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:24.673820 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:24.674498 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:24.675217 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:24.676379 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:24.676839 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:24.676937 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:24.680624 (+ 3687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.129:51839: BlockingRecv error: recv got EOF from 127.31.235.129:51839 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 20:23:25 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:25 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:25.126099 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:25.118024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.130:42009)
0620 20:23:25.118630 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:25.118651 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:25.118762 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:25.119119 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:25.119123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:25.119558 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:25.119707 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:25.119715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:25.120710 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:25.120715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:25.121655 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:25.121668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:25.122291 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:25.122296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:25.122457 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:25.123029 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:25.123045 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:25.125873 (+ 2828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":385,"threads_started":1}
W20260620 20:23:25.126258 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:25.118080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:42009 (local address 127.31.235.190:35007)
0620 20:23:25.118295 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:25.118300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:25.118730 (+ 430us) server_negotiation.cc:407] Connection header received
0620 20:23:25.118856 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:25.118860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:25.118935 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:25.119063 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:25.119836 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:25.120561 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:25.121771 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:25.122164 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:25.122251 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:25.126060 (+ 3809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.130:42009: BlockingRecv error: recv got EOF from 127.31.235.130:42009 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 20:23:25 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:25 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:25.581254 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:25.572877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.131:46545)
0620 20:23:25.573438 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:25.573454 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:25.573595 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:25.573874 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:25.573878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:25.574261 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:25.574439 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:25.574448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:25.575413 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:25.575418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:25.576358 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:25.576370 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:25.577047 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:25.577051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:25.577178 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:25.577827 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:25.577851 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:25.580929 (+ 3078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":328,"threads_started":1}
W20260620 20:23:25.581434 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:25.572902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:46545 (local address 127.31.235.190:35007)
0620 20:23:25.573064 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:25.573068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:25.573544 (+ 476us) server_negotiation.cc:407] Connection header received
0620 20:23:25.573688 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:25.573691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:25.573750 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:25.573827 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:25.574561 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:25.575263 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:25.576493 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:25.576918 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:25.577010 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:25.581162 (+ 4152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.131:46545: BlockingRecv error: recv got EOF from 127.31.235.131:46545 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 20:23:25 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:25 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:25.689791 9505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:25.681706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.129:47431)
0620 20:23:25.682257 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:25.682273 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:25.682389 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:25.682677 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:25.682680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:25.683080 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:25.683230 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:25.683239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:25.684222 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:25.684227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:25.685180 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:25.685193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:25.685916 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:25.685920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:25.686038 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:25.686636 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:25.686653 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:25.689527 (+ 2874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":292,"threads_started":1}
W20260620 20:23:25.690004 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:25.681777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:47431 (local address 127.31.235.190:35007)
0620 20:23:25.681969 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:25.681973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:25.682343 (+ 370us) server_negotiation.cc:407] Connection header received
0620 20:23:25.682487 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:25.682490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:25.682550 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:25.682630 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:25.683356 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:25.684075 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:25.685304 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:25.685735 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:25.685825 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:25.689766 (+ 3941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.129:47431: BlockingRecv error: recv got EOF from 127.31.235.129:47431 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 20:23:26 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:26 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:26.136768 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:26.127414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.130:34699)
0620 20:23:26.128000 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:26.128021 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:26.128188 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:26.128556 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:26.128561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:26.129028 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:26.129194 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:26.129204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:26.130397 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:26.130405 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:26.131425 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:26.131437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:26.132238 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:26.132245 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:26.132434 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:26.133108 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:26.133126 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:26.136282 (+ 3156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":289,"threads_started":1}
W20260620 20:23:26.136905 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:26.127570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:34699 (local address 127.31.235.190:35007)
0620 20:23:26.127774 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:26.127783 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:26.128115 (+ 332us) server_negotiation.cc:407] Connection header received
0620 20:23:26.128343 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:26.128353 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:26.128417 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:26.128526 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:26.129350 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:26.130240 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:26.131580 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:26.132038 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:26.132195 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:26.136509 (+ 4314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.130:34699: BlockingRecv error: recv got EOF from 127.31.235.130:34699 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 20:23:26 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:26 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:26.590709 9507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:26.582350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.131:58623)
0620 20:23:26.582878 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:26.582894 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:26.583072 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:26.583398 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:26.583402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:26.583813 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:26.583967 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:26.583976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:26.584977 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:26.584986 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:26.585931 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:26.585942 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:26.586631 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:26.586636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:26.586801 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:26.587415 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:26.587432 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:26.590339 (+ 2907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":288,"threads_started":1}
W20260620 20:23:26.590740 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:26.582459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:58623 (local address 127.31.235.190:35007)
0620 20:23:26.582702 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:26.582709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:26.583000 (+ 291us) server_negotiation.cc:407] Connection header received
0620 20:23:26.583208 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:26.583212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:26.583269 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:26.583370 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:26.584104 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:26.584830 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:26.586078 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:26.586488 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:26.586621 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:26.590531 (+ 3910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.131:58623: BlockingRecv error: recv got EOF from 127.31.235.131:58623 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 20:23:26 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:26 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:26.699180 9508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:26.690837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.129:36343)
0620 20:23:26.691352 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:26.691368 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:26.691488 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:26.691805 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:26.691808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:26.692145 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:26.692318 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:26.692325 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:26.693298 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:26.693302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:26.694294 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:26.694307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:26.694983 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:26.694987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:26.695101 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:26.695755 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:26.695772 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:26.698943 (+ 3171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":295,"threads_started":1}
W20260620 20:23:26.699486 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:26.690899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:36343 (local address 127.31.235.190:35007)
0620 20:23:26.691150 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:26.691154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:26.691464 (+ 310us) server_negotiation.cc:407] Connection header received
0620 20:23:26.691613 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:26.691616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:26.691675 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:26.691757 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:26.692449 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:26.693172 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:26.694423 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:26.694860 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:26.694955 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:26.699191 (+ 4236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.129:36343: BlockingRecv error: recv got EOF from 127.31.235.129:36343 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Jun 20 20:23:27 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:27 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:27.146183 9509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:27.137844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.130:58521)
0620 20:23:27.138425 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:27.138446 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:27.138557 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:27.138849 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:27.138852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:27.139200 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:27.139383 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:27.139391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:27.140371 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:27.140374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:27.141319 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:27.141334 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:27.142290 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:27.142323 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:27.142500 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:27.143192 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:27.143210 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:27.145943 (+ 2733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":343,"threads_started":1}
W20260620 20:23:27.146345 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:27.137945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:58521 (local address 127.31.235.190:35007)
0620 20:23:27.138125 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:27.138130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:27.138520 (+ 390us) server_negotiation.cc:407] Connection header received
0620 20:23:27.138653 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:27.138657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:27.138717 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:27.138798 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:27.139512 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:27.140223 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:27.141460 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:27.142129 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:27.142285 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:27.146127 (+ 3842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.130:58521: BlockingRecv error: recv got EOF from 127.31.235.130:58521 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 20:23:27 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:27 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.131@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:27.603703 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:27.591779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.131:49387)
0620 20:23:27.592362 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:27.592379 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:27.592509 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:27.592806 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:27.592809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:27.593137 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:27.593330 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:27.593339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:27.594363 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:27.594368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:27.595296 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:27.595308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:27.596060 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:27.596065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:27.596203 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:27.600407 (+ 4204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:27.600426 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:27.603460 (+ 3034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":327,"threads_started":1}
W20260620 20:23:27.603863 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:27.591806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:49387 (local address 127.31.235.190:35007)
0620 20:23:27.591999 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:27.592003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:27.592468 (+ 465us) server_negotiation.cc:407] Connection header received
0620 20:23:27.592606 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:27.592609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:27.592667 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:27.592743 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:27.593465 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:27.594191 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:27.595452 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:27.595901 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:27.596003 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:27.603654 (+ 7651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.131:49387: BlockingRecv error: recv got EOF from 127.31.235.131:49387 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 20:23:27 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:27 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.129@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:27.708807 9511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:27.700482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.129:54629)
0620 20:23:27.701053 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:27.701069 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:27.701176 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:27.701601 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:27.701606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:27.702052 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:27.702229 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:27.702240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:27.703276 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:27.703283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:27.704249 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:27.704262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:27.704953 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:27.704958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:27.705077 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:27.705768 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:27.705786 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:27.708566 (+ 2780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":315,"threads_started":1}
W20260620 20:23:27.708983 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:27.700533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:54629 (local address 127.31.235.190:35007)
0620 20:23:27.700710 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:27.700714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:27.701142 (+ 428us) server_negotiation.cc:407] Connection header received
0620 20:23:27.701273 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:27.701277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:27.701354 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:27.701430 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:27.702381 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:27.703108 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:27.704383 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:27.704789 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:27.704882 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:27.708737 (+ 3855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.129:54629: BlockingRecv error: recv got EOF from 127.31.235.129:54629 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 20:23:28 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 20:23:28 dist-test-slave-zzwf krb5kdc[8467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.235.130@KRBTEST.COM for oryx/127.31.235.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 20:23:28.156006 9512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:28.147286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35007 (local address 127.31.235.130:60285)
0620 20:23:28.147845 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:28.147865 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:28.148022 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:28.148395 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:28.148400 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:28.148803 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:28.148954 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:28.148962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:28.150113 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:28.150122 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:28.151159 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:28.151173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:28.151978 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:28.151984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:28.152157 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:28.152754 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:28.152771 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:28.155757 (+ 2986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.235.190:35007: Server oryx/127.31.235.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":306,"threads_started":1}
W20260620 20:23:28.156194 9203 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:28.147405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:60285 (local address 127.31.235.190:35007)
0620 20:23:28.147651 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:28.147659 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:28.147955 (+ 296us) server_negotiation.cc:407] Connection header received
0620 20:23:28.148170 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:28.148174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:28.148233 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:28.148355 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:28.149086 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:28.149889 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:28.151326 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:28.151821 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:28.151970 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:28.155944 (+ 3974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.235.130:60285: BlockingRecv error: recv got EOF from 127.31.235.130:60285 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:23:28.432569 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 9054
I20260620 20:23:28.443993 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:28.433697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46652 (local address 127.31.235.129:40821)
0620 20:23:28.434230 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:28.434237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:28.434262 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:23:28.434317 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:28.434323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:28.434607 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:28.434753 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:28.436113 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:28.437122 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:28.438297 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:28.438822 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:28.438926 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:28.441286 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:28.441294 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:28.441307 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:28.441339 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:28.443501 (+ 2162us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.235.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 20:23:28.443729 (+ 228us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46652: Request ticket server kudu/127.31.235.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":280,"threads_started":1}
W20260620 20:23:28.444093 9514 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46652: Request ticket server kudu/127.31.235.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 20:23:28.444105 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 9054: Not authorized: Client connection negotiation failed: client connection to 127.31.235.129:40821: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.235.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 20:23:28.444326 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 9054
I20260620 20:23:28.460685 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 9205
I20260620 20:23:28.472504 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:28.461718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40054 (local address 127.31.235.130:34993)
0620 20:23:28.462287 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:28.462294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:28.462322 (+ 28us) server_negotiation.cc:407] Connection header received
0620 20:23:28.462377 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:28.462384 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:28.462615 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:28.462756 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:28.463992 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:28.465129 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:28.466514 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:28.467036 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:28.467137 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:28.469597 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:28.469605 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:28.469619 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:28.469650 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:28.472040 (+ 2390us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.235.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 20:23:28.472272 (+ 232us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40054: Request ticket server kudu/127.31.235.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":277,"threads_started":1}
W20260620 20:23:28.472633 9515 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40054: Request ticket server kudu/127.31.235.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 20:23:28.472826 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 9205: Not authorized: Client connection negotiation failed: client connection to 127.31.235.130:34993: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.235.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 20:23:28.473001 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 9205
I20260620 20:23:28.488891 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 9336
I20260620 20:23:28.500669 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:28.489937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55918 (local address 127.31.235.131:36943)
0620 20:23:28.490486 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:28.490493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:28.490518 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:23:28.490582 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:28.490589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:28.490806 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:28.490953 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:28.492209 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:28.493229 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:28.494437 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:28.494940 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:28.495033 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:28.497456 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:28.497482 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:28.497560 (+ 78us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:28.497589 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:28.500075 (+ 2486us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.235.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 20:23:28.500371 (+ 296us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55918: Request ticket server kudu/127.31.235.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":288,"threads_started":1}
W20260620 20:23:28.500767 9516 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55918: Request ticket server kudu/127.31.235.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 20:23:28.500809 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 9336: Not authorized: Client connection negotiation failed: client connection to 127.31.235.131:36943: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.31.235.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 20:23:28.500994 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 9336
I20260620 20:23:28.517468 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 8985
I20260620 20:23:28.518767 9047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:23:28.632066 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 8985
2026-06-20T20:23:28Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21105 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T20:23:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:23:28Z Disabled control of system clock
I20260620 20:23:28.712018 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:40967
--webserver_interface=127.31.235.190
--webserver_port=0
--builtin_ntp_servers=127.31.235.148:43189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:40967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 20:23:29.040791 9521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:29.041078 9521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:29.041157 9521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:29.048759 9521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:23:29.048877 9521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:29.048914 9521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:29.048964 9521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:23:29.049008 9521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:23:29.058272 9521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:40967
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:40967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:29.059743 9521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:29.061635 9521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:29.074139 9529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:29.074198 9526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:29.074112 9527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:29.074456 9521 server_base.cc:1034] running on GCE node
I20260620 20:23:29.075500 9521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:29.076926 9521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:29.078152 9521 hybrid_clock.cc:648] HybridClock initialized: now 1781987009078093 us; error 53 us; skew 500 ppm
I20260620 20:23:29.078517 9521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:23:29.080747 9521 webserver.cc:460] Webserver started at http://127.31.235.190:34135/ using document root <none> and password file <none>
I20260620 20:23:29.081462 9521 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:29.081650 9521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:29.081995 9521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:23:29.084873 9521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "916b4585f6cc4017b01e3ca6dc68b005"
format_stamp: "Formatted at 2026-06-20 20:23:29 on dist-test-slave-zzwf"
server_key: "6f91d6a40ff2dea5157914585dae6a4f"
server_key_iv: "9a9c1aa7b07fed197e8c18059b983654"
server_key_version: "encryptionkey@0"
I20260620 20:23:29.085753 9521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "916b4585f6cc4017b01e3ca6dc68b005"
format_stamp: "Formatted at 2026-06-20 20:23:29 on dist-test-slave-zzwf"
server_key: "6f91d6a40ff2dea5157914585dae6a4f"
server_key_iv: "9a9c1aa7b07fed197e8c18059b983654"
server_key_version: "encryptionkey@0"
I20260620 20:23:29.091090 9521 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 20:23:29.094657 9535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:29.096096 9521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:23:29.096292 9521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "916b4585f6cc4017b01e3ca6dc68b005"
format_stamp: "Formatted at 2026-06-20 20:23:29 on dist-test-slave-zzwf"
server_key: "6f91d6a40ff2dea5157914585dae6a4f"
server_key_iv: "9a9c1aa7b07fed197e8c18059b983654"
server_key_version: "encryptionkey@0"
I20260620 20:23:29.096517 9521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:29.132789 9521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:29.133751 9521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:29.134076 9521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:29.152535 9521 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:40967
I20260620 20:23:29.152575 9586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:40967 every 8 connection(s)
I20260620 20:23:29.153908 9521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:23:29.154785 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 9521
I20260620 20:23:29.154980 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:23:29.155753 32686 external_mini_cluster.cc:1428] Setting key 45bbfc8e25d8f48f3f533e7277844065
I20260620 20:23:29.158619 9587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:29.169417 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:29.157901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54664 (local address 127.31.235.190:40967)
0620 20:23:29.158954 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:29.158965 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:29.159006 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:23:29.160536 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:29.160555 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:29.160977 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:29.161286 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:29.161923 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:29.163211 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:29.165137 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:29.165841 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:29.165961 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:29.167206 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:29.167212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:29.167228 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:23:29.167771 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:29.167846 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:23:29.167865 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:29.167959 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:29.168177 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 20:23:29.168535 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":398,"threads_started":1}
I20260620 20:23:29.170889 9587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005: Bootstrap starting.
I20260620 20:23:29.173926 9587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:29.174892 9587 log.cc:824] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005: Log is configured to *not* fsync() on all Append() calls
I20260620 20:23:29.177629 9587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005: No bootstrap required, opened a new log
I20260620 20:23:29.184648 9587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916b4585f6cc4017b01e3ca6dc68b005" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 40967 } }
I20260620 20:23:29.185016 9587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:29.185128 9587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916b4585f6cc4017b01e3ca6dc68b005, State: Initialized, Role: FOLLOWER
I20260620 20:23:29.185740 9587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916b4585f6cc4017b01e3ca6dc68b005" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 40967 } }
I20260620 20:23:29.185984 9587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:23:29.186121 9587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:23:29.186300 9587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:29.188539 9587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916b4585f6cc4017b01e3ca6dc68b005" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 40967 } }
I20260620 20:23:29.189044 9587 leader_election.cc:302] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 916b4585f6cc4017b01e3ca6dc68b005; no voters:
I20260620 20:23:29.189736 9587 leader_election.cc:288] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:23:29.189829 9592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:23:29.191170 9592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005 [term 1 LEADER]: Becoming Leader. State: Replica: 916b4585f6cc4017b01e3ca6dc68b005, State: Running, Role: LEADER
I20260620 20:23:29.191658 9592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916b4585f6cc4017b01e3ca6dc68b005" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 40967 } }
I20260620 20:23:29.191910 9587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:23:29.194440 9594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 916b4585f6cc4017b01e3ca6dc68b005. Latest consensus state: current_term: 1 leader_uuid: "916b4585f6cc4017b01e3ca6dc68b005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916b4585f6cc4017b01e3ca6dc68b005" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 40967 } } }
I20260620 20:23:29.194569 9593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "916b4585f6cc4017b01e3ca6dc68b005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916b4585f6cc4017b01e3ca6dc68b005" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 40967 } } }
I20260620 20:23:29.194844 9593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005 [sys.catalog]: This master's current role is: LEADER
I20260620 20:23:29.194849 9594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005 [sys.catalog]: This master's current role is: LEADER
I20260620 20:23:29.196481 9601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:23:29.201459 9601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:23:29.209971 9601 catalog_manager.cc:1347] Generated new cluster ID: 056a0c60a3354d5aae999762e6524973
I20260620 20:23:29.210069 9601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:23:29.224645 9601 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:23:29.226300 9601 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:23:29.233984 9601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 916b4585f6cc4017b01e3ca6dc68b005: Generated new TSK 0
I20260620 20:23:29.234905 9601 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:23:29.240594 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:40967
--builtin_ntp_servers=127.31.235.148:43189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 20:23:29.569988 9611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:29.570257 9611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:29.570377 9611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:29.577548 9611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:29.577661 9611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:29.577765 9611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:23:29.586994 9611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:40967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:29.588173 9611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:29.589875 9611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:29.603391 9616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:29.603430 9617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:29.603406 9619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:29.604103 9611 server_base.cc:1034] running on GCE node
I20260620 20:23:29.604686 9611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:29.605681 9611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:29.606904 9611 hybrid_clock.cc:648] HybridClock initialized: now 1781987009606842 us; error 49 us; skew 500 ppm
I20260620 20:23:29.607215 9611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:23:29.609480 9611 webserver.cc:460] Webserver started at http://127.31.235.129:40609/ using document root <none> and password file <none>
I20260620 20:23:29.610244 9611 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:29.610379 9611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:29.610693 9611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:23:29.613754 9611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "677255b3b4a0438096aea58e71186cbb"
format_stamp: "Formatted at 2026-06-20 20:23:29 on dist-test-slave-zzwf"
server_key: "ca41b747aea5f602aa735901c899112e"
server_key_iv: "abe4d8ea84c6faaea5e50510bbcf642b"
server_key_version: "encryptionkey@0"
I20260620 20:23:29.614557 9611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "677255b3b4a0438096aea58e71186cbb"
format_stamp: "Formatted at 2026-06-20 20:23:29 on dist-test-slave-zzwf"
server_key: "ca41b747aea5f602aa735901c899112e"
server_key_iv: "abe4d8ea84c6faaea5e50510bbcf642b"
server_key_version: "encryptionkey@0"
I20260620 20:23:29.619596 9611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:23:29.623009 9625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:29.624327 9611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:23:29.624531 9611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "677255b3b4a0438096aea58e71186cbb"
format_stamp: "Formatted at 2026-06-20 20:23:29 on dist-test-slave-zzwf"
server_key: "ca41b747aea5f602aa735901c899112e"
server_key_iv: "abe4d8ea84c6faaea5e50510bbcf642b"
server_key_version: "encryptionkey@0"
I20260620 20:23:29.624748 9611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:29.645256 9611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:29.646152 9611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:29.646458 9611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:29.647416 9611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:23:29.648902 9611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:23:29.649024 9611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:29.649164 9611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:23:29.649252 9611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:29.682747 9611 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:44507
I20260620 20:23:29.682837 9737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:44507 every 8 connection(s)
I20260620 20:23:29.684032 9611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:23:29.692899 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 9611
I20260620 20:23:29.693081 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:23:29.693897 32686 external_mini_cluster.cc:1428] Setting key e06b9d6d848fdc288059732be2b33b04
I20260620 20:23:29.696807 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:40967
--builtin_ntp_servers=127.31.235.148:43189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 20:23:29.698124 9740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:29.686877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:40763 (local address 127.31.235.190:40967)
0620 20:23:29.687325 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:29.687329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:29.688524 (+ 1195us) server_negotiation.cc:407] Connection header received
0620 20:23:29.690352 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:29.690358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:29.690449 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:29.690566 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:29.691603 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:29.692407 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:29.694573 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:29.695122 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:29.695319 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:29.697086 (+ 1767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:29.697094 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:29.697102 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:23:29.697124 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:29.697181 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:23:29.697198 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:29.697287 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:29.697569 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 20:23:29.697677 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":243,"threads_started":1}
I20260620 20:23:29.698933 9741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:29.687463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:40967 (local address 127.31.235.129:40763)
0620 20:23:29.688338 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:29.688371 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:29.690090 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:29.690795 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:29.690803 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:29.690888 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:29.691439 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:29.691456 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:29.692613 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:29.692618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:29.694384 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:29.694425 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:29.695478 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:29.695486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:29.695847 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:29.696659 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:23:29.696663 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:29.696721 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:23:29.696850 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:23:29.697298 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:23:29.697307 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:23:29.697314 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 20:23:29.697559 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 20:23:29.697848 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":290,"threads_started":1}
I20260620 20:23:29.700476 9738 heartbeater.cc:347] Connected to a master server at 127.31.235.190:40967
I20260620 20:23:29.700783 9738 heartbeater.cc:464] Registering TS with master...
I20260620 20:23:29.701717 9738 heartbeater.cc:511] Master 127.31.235.190:40967 requested a full tablet report, sending...
I20260620 20:23:29.703924 9552 ts_manager.cc:194] Registered new tserver with Master: 677255b3b4a0438096aea58e71186cbb (127.31.235.129:44507)
I20260620 20:23:29.705402 9552 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.235.129:40763
W20260620 20:23:30.029017 9742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:30.029301 9742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:30.029443 9742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:30.036808 9742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:30.036921 9742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:30.037093 9742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:23:30.046448 9742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:40967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:30.047765 9742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:30.049643 9742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:30.063158 9748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:30.063184 9750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:30.063413 9747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:30.064064 9742 server_base.cc:1034] running on GCE node
I20260620 20:23:30.064643 9742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:30.065675 9742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:30.066915 9742 hybrid_clock.cc:648] HybridClock initialized: now 1781987010066844 us; error 55 us; skew 500 ppm
I20260620 20:23:30.067232 9742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:23:30.069860 9742 webserver.cc:460] Webserver started at http://127.31.235.130:39299/ using document root <none> and password file <none>
I20260620 20:23:30.070595 9742 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:30.070694 9742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:30.070967 9742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:23:30.073884 9742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "f55856f781d342d2b832f108bce06692"
format_stamp: "Formatted at 2026-06-20 20:23:30 on dist-test-slave-zzwf"
server_key: "d7db7e2fa8ab17a7f4f7362882c75030"
server_key_iv: "24ecbcbaa1c808a6daedd9d1e0dcc60b"
server_key_version: "encryptionkey@0"
I20260620 20:23:30.074671 9742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "f55856f781d342d2b832f108bce06692"
format_stamp: "Formatted at 2026-06-20 20:23:30 on dist-test-slave-zzwf"
server_key: "d7db7e2fa8ab17a7f4f7362882c75030"
server_key_iv: "24ecbcbaa1c808a6daedd9d1e0dcc60b"
server_key_version: "encryptionkey@0"
I20260620 20:23:30.079990 9742 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 20:23:30.083659 9756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:30.085153 9742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:23:30.085302 9742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "f55856f781d342d2b832f108bce06692"
format_stamp: "Formatted at 2026-06-20 20:23:30 on dist-test-slave-zzwf"
server_key: "d7db7e2fa8ab17a7f4f7362882c75030"
server_key_iv: "24ecbcbaa1c808a6daedd9d1e0dcc60b"
server_key_version: "encryptionkey@0"
I20260620 20:23:30.085613 9742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:30.103346 9742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:30.104142 9742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:30.104405 9742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:30.105288 9742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:23:30.106750 9742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:23:30.106837 9742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:30.106914 9742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:23:30.106971 9742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:30.139922 9742 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:43155
I20260620 20:23:30.140015 9868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:43155 every 8 connection(s)
I20260620 20:23:30.141218 9742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:23:30.151502 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 9742
I20260620 20:23:30.151702 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:23:30.152650 32686 external_mini_cluster.cc:1428] Setting key fdf1540582813d8ddedd1c02a8ed7a1a
I20260620 20:23:30.153841 9740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:30.143662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:51147 (local address 127.31.235.190:40967)
0620 20:23:30.143864 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:30.143870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:30.145307 (+ 1437us) server_negotiation.cc:407] Connection header received
0620 20:23:30.147082 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:30.147086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:30.147150 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:30.147239 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:30.148251 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:30.149106 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:30.150660 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:30.151083 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:30.151189 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:30.152884 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:30.152890 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:30.152894 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:23:30.152916 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:30.152957 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:23:30.152973 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:30.153067 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:30.153381 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 20:23:30.153519 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:23:30.155287 9871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:30.144242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:40967 (local address 127.31.235.130:51147)
0620 20:23:30.145128 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:30.145185 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:30.146803 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:30.147440 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:30.147448 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:30.147531 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:30.148092 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:30.148107 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:30.149241 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:30.149245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:30.150518 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:30.150529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:30.151312 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:30.151320 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:30.151630 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:30.152444 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:23:30.152450 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:30.152503 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:23:30.152598 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:23:30.153129 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:23:30.153140 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:23:30.153148 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 20:23:30.153561 (+ 413us) client_negotiation.cc:240] Negotiation successful
0620 20:23:30.153964 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":254,"threads_started":1}
I20260620 20:23:30.155701 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:40967
--builtin_ntp_servers=127.31.235.148:43189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 20:23:30.157124 9869 heartbeater.cc:347] Connected to a master server at 127.31.235.190:40967
I20260620 20:23:30.157451 9869 heartbeater.cc:464] Registering TS with master...
I20260620 20:23:30.158356 9869 heartbeater.cc:511] Master 127.31.235.190:40967 requested a full tablet report, sending...
I20260620 20:23:30.159951 9552 ts_manager.cc:194] Registered new tserver with Master: f55856f781d342d2b832f108bce06692 (127.31.235.130:43155)
I20260620 20:23:30.161170 9552 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.235.130:51147
W20260620 20:23:30.488669 9872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:30.488986 9872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:30.489102 9872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:30.496526 9872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:30.496623 9872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:30.496800 9872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:23:30.505957 9872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:40967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:30.507189 9872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:30.508916 9872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:30.521843 9877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:30.521938 9880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:30.521941 9878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:30.527207 9872 server_base.cc:1034] running on GCE node
I20260620 20:23:30.527897 9872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:30.528906 9872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:30.530109 9872 hybrid_clock.cc:648] HybridClock initialized: now 1781987010530063 us; error 50 us; skew 500 ppm
I20260620 20:23:30.530460 9872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:23:30.532684 9872 webserver.cc:460] Webserver started at http://127.31.235.131:35709/ using document root <none> and password file <none>
I20260620 20:23:30.533386 9872 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:30.533524 9872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:30.533856 9872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:23:30.536644 9872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "b665dd61cca94eb6a63c4c1de997fb47"
format_stamp: "Formatted at 2026-06-20 20:23:30 on dist-test-slave-zzwf"
server_key: "bc19d0c94737052ded37e37c40af282b"
server_key_iv: "0d5e1b6afa91ce942239b018b5167ded"
server_key_version: "encryptionkey@0"
I20260620 20:23:30.537432 9872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "b665dd61cca94eb6a63c4c1de997fb47"
format_stamp: "Formatted at 2026-06-20 20:23:30 on dist-test-slave-zzwf"
server_key: "bc19d0c94737052ded37e37c40af282b"
server_key_iv: "0d5e1b6afa91ce942239b018b5167ded"
server_key_version: "encryptionkey@0"
I20260620 20:23:30.542517 9872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 20:23:30.545959 9886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:30.547323 9872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 20:23:30.547515 9872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "b665dd61cca94eb6a63c4c1de997fb47"
format_stamp: "Formatted at 2026-06-20 20:23:30 on dist-test-slave-zzwf"
server_key: "bc19d0c94737052ded37e37c40af282b"
server_key_iv: "0d5e1b6afa91ce942239b018b5167ded"
server_key_version: "encryptionkey@0"
I20260620 20:23:30.547740 9872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:30.581773 9872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:30.582687 9872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:30.582993 9872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:30.583966 9872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:23:30.585361 9872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:23:30.585458 9872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:30.585614 9872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:23:30.585678 9872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:30.618819 9872 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:39861
I20260620 20:23:30.618873 9998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:39861 every 8 connection(s)
I20260620 20:23:30.620100 9872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:23:30.624061 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 9872
I20260620 20:23:30.624228 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:23:30.624821 32686 external_mini_cluster.cc:1428] Setting key 9633fae36d1d2f07c71dc9566a850201
I20260620 20:23:30.633555 9740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:30.622602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:44593 (local address 127.31.235.190:40967)
0620 20:23:30.622779 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:30.622783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:30.624476 (+ 1693us) server_negotiation.cc:407] Connection header received
0620 20:23:30.626921 (+ 2445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:30.626929 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:30.627030 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:30.627138 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:30.628487 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:30.629310 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:30.630995 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:30.631405 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:30.631526 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:30.632818 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:30.632823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:30.632826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:23:30.632843 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:30.632873 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:23:30.632894 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:30.632965 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:30.633164 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 20:23:30.633234 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:23:30.634555 10001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:30.623139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:40967 (local address 127.31.235.131:44593)
0620 20:23:30.624277 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:30.624342 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:30.626588 (+ 2246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:30.627380 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:30.627388 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:30.627470 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:30.628273 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:30.628299 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:30.629520 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:30.629525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:30.630837 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:30.630852 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:30.631647 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:30.631652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:30.631850 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:30.632530 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:23:30.632533 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:30.632571 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:23:30.632645 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:23:30.632988 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:23:30.632995 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:23:30.633000 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:23:30.633138 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 20:23:30.633418 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":305,"threads_started":1}
I20260620 20:23:30.635773 9999 heartbeater.cc:347] Connected to a master server at 127.31.235.190:40967
I20260620 20:23:30.636073 9999 heartbeater.cc:464] Registering TS with master...
I20260620 20:23:30.636785 9999 heartbeater.cc:511] Master 127.31.235.190:40967 requested a full tablet report, sending...
I20260620 20:23:30.638275 9552 ts_manager.cc:194] Registered new tserver with Master: b665dd61cca94eb6a63c4c1de997fb47 (127.31.235.131:39861)
I20260620 20:23:30.639250 9552 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.235.131:44593
I20260620 20:23:30.640503 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 20:23:30.646636 9740 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:30.644917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54680 (local address 127.31.235.190:40967)
0620 20:23:30.645066 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:30.645070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:30.645627 (+ 557us) server_negotiation.cc:407] Connection header received
0620 20:23:30.645811 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:30.645816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:30.645897 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:30.645997 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:30.646388 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54680: BlockingRecv error: recv got EOF from 127.0.0.1:54680 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:23:30.647197 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 9611
I20260620 20:23:30.654947 10008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:30.648090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46618 (local address 127.31.235.129:44507)
0620 20:23:30.648608 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:30.648614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:30.648637 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:23:30.648686 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:30.648692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:30.648888 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:30.649052 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:30.649413 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:30.650417 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:30.651722 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:30.652330 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:30.652465 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:30.653535 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:30.653550 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:30.653563 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:23:30.654132 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:30.654175 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:23:30.654193 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:30.654316 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:30.654478 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 20:23:30.654616 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":264,"threads_started":1}
I20260620 20:23:30.656206 9733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:23:30.736318 9738 heartbeater.cc:503] Master 127.31.235.190:40967 was elected leader, sending a full tablet report...
I20260620 20:23:30.737447 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 9611
W20260620 20:23:30.753870 9536 connection.cc:535] server connection from 127.31.235.129:40763 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 20:23:30.754236 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 9742
I20260620 20:23:30.762024 10010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:30.755149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59574 (local address 127.31.235.130:43155)
0620 20:23:30.755699 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:30.755706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:30.755733 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:23:30.755795 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:30.755801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:30.756011 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:30.756135 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:30.756533 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:30.757678 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:30.758873 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:30.759416 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:30.759519 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:30.760534 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:30.760542 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:30.760554 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:23:30.761117 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:30.761168 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:23:30.761185 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:30.761272 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:30.761581 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 20:23:30.761711 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":305,"threads_started":1}
I20260620 20:23:30.763306 9864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:23:30.848655 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 9742
I20260620 20:23:30.864676 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 9872
I20260620 20:23:30.873246 10012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:30.865770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49118 (local address 127.31.235.131:39861)
0620 20:23:30.866431 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:30.866439 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:30.866469 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:23:30.866528 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:30.866535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:30.866760 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:30.866900 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:30.867346 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:30.868423 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:30.869824 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:30.870442 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:30.870550 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:30.871710 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:30.871718 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:30.871732 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:23:30.872376 (+ 644us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:30.872423 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:23:30.872443 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:30.872532 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:30.872728 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 20:23:30.872878 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":357,"threads_started":1}
I20260620 20:23:30.874583 9994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:23:30.959362 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 9872
I20260620 20:23:30.977428 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 9521
I20260620 20:23:30.978708 9582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:23:31.080149 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 9521
2026-06-20T20:23:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2443 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T20:23:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:23:31Z Disabled control of system clock
I20260620 20:23:31.150730 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:33029
--webserver_interface=127.31.235.190
--webserver_port=0
--builtin_ntp_servers=127.31.235.148:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:33029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 20:23:31.492144 10018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:31.492460 10018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:31.492547 10018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:31.500085 10018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:23:31.500209 10018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:31.500247 10018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:31.500303 10018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:23:31.500350 10018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:23:31.509927 10018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:33029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:33029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:31.511152 10018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:31.512852 10018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:31.524869 10023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:31.524775 10026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:31.524899 10024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:31.525914 10018 server_base.cc:1034] running on GCE node
I20260620 20:23:31.526741 10018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:31.528127 10018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:31.529343 10018 hybrid_clock.cc:648] HybridClock initialized: now 1781987011529287 us; error 45 us; skew 500 ppm
I20260620 20:23:31.529736 10018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:23:31.531983 10018 webserver.cc:460] Webserver started at http://127.31.235.190:35197/ using document root <none> and password file <none>
I20260620 20:23:31.532684 10018 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:31.532770 10018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:31.533128 10018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:23:31.536000 10018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "3f421bd281ba47d28e0f7e905ae57ba8"
format_stamp: "Formatted at 2026-06-20 20:23:31 on dist-test-slave-zzwf"
server_key: "1d464ec05a11bc5011e9d8f3be4283a9"
server_key_iv: "5d5809e377c36929cf9625ed5a8397ba"
server_key_version: "encryptionkey@0"
I20260620 20:23:31.536767 10018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "3f421bd281ba47d28e0f7e905ae57ba8"
format_stamp: "Formatted at 2026-06-20 20:23:31 on dist-test-slave-zzwf"
server_key: "1d464ec05a11bc5011e9d8f3be4283a9"
server_key_iv: "5d5809e377c36929cf9625ed5a8397ba"
server_key_version: "encryptionkey@0"
I20260620 20:23:31.541778 10018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 20:23:31.545123 10032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:31.546535 10018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:23:31.546722 10018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "3f421bd281ba47d28e0f7e905ae57ba8"
format_stamp: "Formatted at 2026-06-20 20:23:31 on dist-test-slave-zzwf"
server_key: "1d464ec05a11bc5011e9d8f3be4283a9"
server_key_iv: "5d5809e377c36929cf9625ed5a8397ba"
server_key_version: "encryptionkey@0"
I20260620 20:23:31.546955 10018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:31.552623 10018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:31.553403 10018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:31.553719 10018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:31.572407 10018 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:33029
I20260620 20:23:31.572430 10083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:33029 every 8 connection(s)
I20260620 20:23:31.573757 10018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:23:31.577409 10084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:31.583665 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 10018
I20260620 20:23:31.583909 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:23:31.584527 32686 external_mini_cluster.cc:1428] Setting key 376c64ea703b967a3bc3f2d99468a983
I20260620 20:23:31.587594 10084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8: Bootstrap starting.
I20260620 20:23:31.590957 10084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:31.592216 10084 log.cc:824] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8: Log is configured to *not* fsync() on all Append() calls
I20260620 20:23:31.592675 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:31.586479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.31.235.190:33029)
0620 20:23:31.587458 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:31.587469 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:31.587508 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:23:31.589157 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:31.589177 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:31.589632 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:31.589944 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:31.591107 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:31.591114 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:31.591127 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:23:31.591142 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:31.591201 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:23:31.591217 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:31.591409 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:31.591513 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 20:23:31.591793 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":342,"threads_started":1}
I20260620 20:23:31.595526 10084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8: No bootstrap required, opened a new log
I20260620 20:23:31.603008 10084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f421bd281ba47d28e0f7e905ae57ba8" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33029 } }
I20260620 20:23:31.603354 10084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:31.603418 10084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f421bd281ba47d28e0f7e905ae57ba8, State: Initialized, Role: FOLLOWER
I20260620 20:23:31.603905 10084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f421bd281ba47d28e0f7e905ae57ba8" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33029 } }
I20260620 20:23:31.604090 10084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:23:31.604177 10084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:23:31.604300 10084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:31.606441 10084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f421bd281ba47d28e0f7e905ae57ba8" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33029 } }
I20260620 20:23:31.606873 10084 leader_election.cc:302] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f421bd281ba47d28e0f7e905ae57ba8; no voters:
I20260620 20:23:31.607523 10084 leader_election.cc:288] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:23:31.607627 10089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:23:31.609026 10089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8 [term 1 LEADER]: Becoming Leader. State: Replica: 3f421bd281ba47d28e0f7e905ae57ba8, State: Running, Role: LEADER
I20260620 20:23:31.609652 10084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:23:31.609629 10089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f421bd281ba47d28e0f7e905ae57ba8" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33029 } }
I20260620 20:23:31.612756 10091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f421bd281ba47d28e0f7e905ae57ba8. Latest consensus state: current_term: 1 leader_uuid: "3f421bd281ba47d28e0f7e905ae57ba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f421bd281ba47d28e0f7e905ae57ba8" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33029 } } }
I20260620 20:23:31.612800 10090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f421bd281ba47d28e0f7e905ae57ba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f421bd281ba47d28e0f7e905ae57ba8" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33029 } } }
I20260620 20:23:31.613206 10090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8 [sys.catalog]: This master's current role is: LEADER
I20260620 20:23:31.613204 10091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8 [sys.catalog]: This master's current role is: LEADER
I20260620 20:23:31.614106 10099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:23:31.620908 10099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:23:31.629756 10099 catalog_manager.cc:1347] Generated new cluster ID: ec2157911d6441ea957378ac5a8c5b45
I20260620 20:23:31.629873 10099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:23:31.652366 10099 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:23:31.652859 10099 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:23:31.667711 10099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f421bd281ba47d28e0f7e905ae57ba8: Generated new TSK 0
I20260620 20:23:31.668608 10099 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:23:31.679013 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33029
--builtin_ntp_servers=127.31.235.148:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 20:23:32.014267 10108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:32.014554 10108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:32.014700 10108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:32.021966 10108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:32.022078 10108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:32.022249 10108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:23:32.031452 10108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:32.032783 10108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:32.034596 10108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:32.047592 10116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:32.047592 10114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:32.047669 10113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:32.048359 10108 server_base.cc:1034] running on GCE node
I20260620 20:23:32.048940 10108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:32.050037 10108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:32.051307 10108 hybrid_clock.cc:648] HybridClock initialized: now 1781987012051243 us; error 54 us; skew 500 ppm
I20260620 20:23:32.051594 10108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:23:32.053982 10108 webserver.cc:460] Webserver started at http://127.31.235.129:38693/ using document root <none> and password file <none>
I20260620 20:23:32.054623 10108 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:32.054708 10108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:32.054970 10108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:23:32.057760 10108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "4203492f4973414d903091ffea3450f3"
format_stamp: "Formatted at 2026-06-20 20:23:32 on dist-test-slave-zzwf"
server_key: "6b9e090b864846a717773311fb45fbdb"
server_key_iv: "3a02f7e0dbc5dbadda8949d73ea7ae17"
server_key_version: "encryptionkey@0"
I20260620 20:23:32.058480 10108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "4203492f4973414d903091ffea3450f3"
format_stamp: "Formatted at 2026-06-20 20:23:32 on dist-test-slave-zzwf"
server_key: "6b9e090b864846a717773311fb45fbdb"
server_key_iv: "3a02f7e0dbc5dbadda8949d73ea7ae17"
server_key_version: "encryptionkey@0"
I20260620 20:23:32.063658 10108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:23:32.067157 10122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:32.068572 10108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:23:32.068753 10108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "4203492f4973414d903091ffea3450f3"
format_stamp: "Formatted at 2026-06-20 20:23:32 on dist-test-slave-zzwf"
server_key: "6b9e090b864846a717773311fb45fbdb"
server_key_iv: "3a02f7e0dbc5dbadda8949d73ea7ae17"
server_key_version: "encryptionkey@0"
I20260620 20:23:32.068972 10108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:32.074946 10108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:32.075752 10108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:32.076031 10108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:32.076979 10108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:23:32.078460 10108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:23:32.078581 10108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:32.078720 10108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:23:32.078810 10108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:32.112254 10108 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:46643
I20260620 20:23:32.112362 10234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:46643 every 8 connection(s)
I20260620 20:23:32.113893 10108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:23:32.114440 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 10108
I20260620 20:23:32.114681 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:23:32.115506 32686 external_mini_cluster.cc:1428] Setting key 41b42321ac626c8d3d5d193bd16fd1f1
I20260620 20:23:32.119009 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33029
--builtin_ntp_servers=127.31.235.148:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 20:23:32.124111 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:32.116627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:34309 (local address 127.31.235.190:33029)
0620 20:23:32.117926 (+ 1299us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:32.117936 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:32.118955 (+ 1019us) server_negotiation.cc:407] Connection header received
0620 20:23:32.121720 (+ 2765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:32.121730 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:32.121812 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:32.121935 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:32.123444 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:32.123450 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:32.123453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:23:32.123469 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:32.123512 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:23:32.123524 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:32.123625 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:32.123846 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 20:23:32.123897 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1140,"thread_start_us":1014,"threads_started":1}
I20260620 20:23:32.125422 10238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:32.117337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33029 (local address 127.31.235.129:34309)
0620 20:23:32.118790 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:32.118838 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:32.121382 (+ 2544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:32.122191 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:32.122206 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:32.122338 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:32.123058 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:23:32.123064 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:32.123114 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:23:32.123223 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:23:32.123630 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:23:32.123642 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:23:32.123653 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 20:23:32.123923 (+ 270us) client_negotiation.cc:240] Negotiation successful
0620 20:23:32.124189 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"thread_start_us":663,"threads_started":1}
I20260620 20:23:32.127084 10235 heartbeater.cc:347] Connected to a master server at 127.31.235.190:33029
I20260620 20:23:32.127485 10235 heartbeater.cc:464] Registering TS with master...
I20260620 20:23:32.128057 10235 heartbeater.cc:511] Master 127.31.235.190:33029 requested a full tablet report, sending...
I20260620 20:23:32.130213 10049 ts_manager.cc:194] Registered new tserver with Master: 4203492f4973414d903091ffea3450f3 (127.31.235.129:46643)
W20260620 20:23:32.451171 10239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:32.451522 10239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:32.451704 10239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:32.458936 10239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:32.459034 10239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:32.459216 10239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:23:32.468185 10239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:32.469470 10239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:32.471210 10239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:32.484320 10245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:32.484439 10244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:32.484414 10247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:32.485512 10239 server_base.cc:1034] running on GCE node
I20260620 20:23:32.486141 10239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:32.487147 10239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:32.488449 10239 hybrid_clock.cc:648] HybridClock initialized: now 1781987012488358 us; error 66 us; skew 500 ppm
I20260620 20:23:32.488785 10239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:23:32.491066 10239 webserver.cc:460] Webserver started at http://127.31.235.130:36845/ using document root <none> and password file <none>
I20260620 20:23:32.491745 10239 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:32.491880 10239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:32.492193 10239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:23:32.495076 10239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "407d417c05994d93b5b0b7e820762263"
format_stamp: "Formatted at 2026-06-20 20:23:32 on dist-test-slave-zzwf"
server_key: "dab0145085c8aba0475799b4a9415b37"
server_key_iv: "27e08ec1408f8bfaf031e3b2fff44e1f"
server_key_version: "encryptionkey@0"
I20260620 20:23:32.495843 10239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "407d417c05994d93b5b0b7e820762263"
format_stamp: "Formatted at 2026-06-20 20:23:32 on dist-test-slave-zzwf"
server_key: "dab0145085c8aba0475799b4a9415b37"
server_key_iv: "27e08ec1408f8bfaf031e3b2fff44e1f"
server_key_version: "encryptionkey@0"
I20260620 20:23:32.500824 10239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 20:23:32.504271 10253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:32.505717 10239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:23:32.505887 10239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "407d417c05994d93b5b0b7e820762263"
format_stamp: "Formatted at 2026-06-20 20:23:32 on dist-test-slave-zzwf"
server_key: "dab0145085c8aba0475799b4a9415b37"
server_key_iv: "27e08ec1408f8bfaf031e3b2fff44e1f"
server_key_version: "encryptionkey@0"
I20260620 20:23:32.506103 10239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:32.511912 10239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:32.512755 10239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:32.513031 10239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:32.514206 10239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:23:32.515625 10239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:23:32.515746 10239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:32.515888 10239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:23:32.515978 10239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:32.549702 10239 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:38469
I20260620 20:23:32.549800 10365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:38469 every 8 connection(s)
I20260620 20:23:32.551057 10239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:23:32.554905 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 10239
I20260620 20:23:32.555127 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:23:32.555635 32686 external_mini_cluster.cc:1428] Setting key f09a3e7aafe2818a6d7db39e836b711d
I20260620 20:23:32.558606 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33029
--builtin_ntp_servers=127.31.235.148:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 20:23:32.559916 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:32.553959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:51987 (local address 127.31.235.190:33029)
0620 20:23:32.554270 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:32.554278 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:32.555687 (+ 1409us) server_negotiation.cc:407] Connection header received
0620 20:23:32.557315 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:32.557321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:32.557387 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:32.557543 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:32.559044 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:32.559051 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:32.559057 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:23:32.559084 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:32.559119 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:23:32.559132 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:32.559285 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:32.559534 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 20:23:32.559615 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260620 20:23:32.561000 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:32.554503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33029 (local address 127.31.235.130:51987)
0620 20:23:32.555531 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:32.555574 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:32.557056 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:32.557771 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:32.557780 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:32.557874 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:32.558638 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:23:32.558641 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:32.558682 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:23:32.558838 (+ 156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:23:32.559243 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:23:32.559257 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:23:32.559268 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 20:23:32.559629 (+ 361us) client_negotiation.cc:240] Negotiation successful
0620 20:23:32.559930 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":320,"threads_started":1}
I20260620 20:23:32.566654 10366 heartbeater.cc:347] Connected to a master server at 127.31.235.190:33029
I20260620 20:23:32.567054 10366 heartbeater.cc:464] Registering TS with master...
I20260620 20:23:32.567730 10366 heartbeater.cc:511] Master 127.31.235.190:33029 requested a full tablet report, sending...
I20260620 20:23:32.569283 10049 ts_manager.cc:194] Registered new tserver with Master: 407d417c05994d93b5b0b7e820762263 (127.31.235.130:38469)
W20260620 20:23:32.886078 10369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:32.886350 10369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:32.886437 10369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:32.894176 10369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:32.894275 10369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:32.894382 10369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:23:32.903952 10369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:33029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:32.905171 10369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:32.906950 10369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:32.920166 10377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:32.920248 10374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:32.920224 10375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:32.921475 10369 server_base.cc:1034] running on GCE node
I20260620 20:23:32.922153 10369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:32.923117 10369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:32.924317 10369 hybrid_clock.cc:648] HybridClock initialized: now 1781987012924279 us; error 42 us; skew 500 ppm
I20260620 20:23:32.924647 10369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:23:32.933024 10369 webserver.cc:460] Webserver started at http://127.31.235.131:34797/ using document root <none> and password file <none>
I20260620 20:23:32.933884 10369 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:32.934005 10369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:32.934393 10369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:23:32.937232 10369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "20b23f5358314e2488c8a1f3f01a0a01"
format_stamp: "Formatted at 2026-06-20 20:23:32 on dist-test-slave-zzwf"
server_key: "f4ad0c6543a8451c45aee26ef95099df"
server_key_iv: "a305cd7b7d4925c6e1a32d366fd1536a"
server_key_version: "encryptionkey@0"
I20260620 20:23:32.938067 10369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "20b23f5358314e2488c8a1f3f01a0a01"
format_stamp: "Formatted at 2026-06-20 20:23:32 on dist-test-slave-zzwf"
server_key: "f4ad0c6543a8451c45aee26ef95099df"
server_key_iv: "a305cd7b7d4925c6e1a32d366fd1536a"
server_key_version: "encryptionkey@0"
I20260620 20:23:32.943078 10369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:23:32.946753 10383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:32.948398 10369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:23:32.948693 10369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "20b23f5358314e2488c8a1f3f01a0a01"
format_stamp: "Formatted at 2026-06-20 20:23:32 on dist-test-slave-zzwf"
server_key: "f4ad0c6543a8451c45aee26ef95099df"
server_key_iv: "a305cd7b7d4925c6e1a32d366fd1536a"
server_key_version: "encryptionkey@0"
I20260620 20:23:32.948966 10369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:32.955196 10369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:32.956022 10369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:32.956296 10369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:32.957252 10369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:23:32.959184 10369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:23:32.959331 10369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 20:23:32.959470 10369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:23:32.959555 10369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:32.993739 10369 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:38137
I20260620 20:23:32.993854 10495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:38137 every 8 connection(s)
I20260620 20:23:32.995103 10369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:23:33.003451 10237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:32.997621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:56407 (local address 127.31.235.190:33029)
0620 20:23:32.997796 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:32.997801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:32.999314 (+ 1513us) server_negotiation.cc:407] Connection header received
0620 20:23:33.001123 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:33.001130 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:33.001213 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:33.001353 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:33.002682 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:33.002688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:33.002693 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:23:33.002719 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:33.002751 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:23:33.002762 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:33.002895 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:33.003101 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 20:23:33.003171 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:23:33.004431 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 10369
I20260620 20:23:33.004536 10498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:32.998136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33029 (local address 127.31.235.131:56407)
0620 20:23:32.999128 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:32.999171 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:33.000838 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:33.001618 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:33.001627 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:33.001722 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:33.002372 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:23:33.002374 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:33.002422 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:23:33.002503 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:23:33.002868 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:23:33.002895 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:23:33.002903 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 20:23:33.003179 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 20:23:33.003413 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":303,"threads_started":1}
I20260620 20:23:33.004627 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:23:33.005067 32686 external_mini_cluster.cc:1428] Setting key de87264f69826f366f84c844d37ab3f5
I20260620 20:23:33.005944 10496 heartbeater.cc:347] Connected to a master server at 127.31.235.190:33029
I20260620 20:23:33.006299 10496 heartbeater.cc:464] Registering TS with master...
I20260620 20:23:33.006942 10496 heartbeater.cc:511] Master 127.31.235.190:33029 requested a full tablet report, sending...
I20260620 20:23:33.008190 10049 ts_manager.cc:194] Registered new tserver with Master: 20b23f5358314e2488c8a1f3f01a0a01 (127.31.235.131:38137)
I20260620 20:23:33.020390 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 20:23:33.026810 10237 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:23:33.024854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.31.235.190:33029)
0620 20:23:33.025090 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:33.025100 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:33.025624 (+ 524us) server_negotiation.cc:407] Connection header received
0620 20:23:33.025802 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:33.025806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:33.025860 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:33.025948 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:33.026293 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58814: BlockingRecv error: recv got EOF from 127.0.0.1:58814 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 20:23:33.027751 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 10108
I20260620 20:23:33.031279 10505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:33.028765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58432 (local address 127.31.235.129:46643)
0620 20:23:33.029241 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:33.029248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:33.029273 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:23:33.029335 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:33.029342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:33.029621 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:33.029750 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:33.030644 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:33.030652 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:33.030664 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:23:33.030679 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:33.030731 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:23:33.030750 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:33.030862 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:33.030952 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 20:23:33.031050 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":221,"threads_started":1}
I20260620 20:23:33.032543 10230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:23:33.114517 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 10108
I20260620 20:23:33.131312 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 10239
I20260620 20:23:33.135484 10507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:33.132397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47256 (local address 127.31.235.130:38469)
0620 20:23:33.133015 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:33.133029 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:33.133058 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:23:33.133124 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:33.133171 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:33.133443 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:33.133653 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:33.134594 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:33.134601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:33.134612 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:23:33.134630 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:33.134672 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:23:33.134687 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:33.134799 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:33.135028 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 20:23:33.135186 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":310,"threads_started":1}
I20260620 20:23:33.136767 10361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:23:33.218067 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 10239
I20260620 20:23:33.234285 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 10369
I20260620 20:23:33.238693 10509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:33.235323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38318 (local address 127.31.235.131:38137)
0620 20:23:33.236071 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:33.236079 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:33.236113 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:23:33.236176 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:33.236183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:33.236413 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:33.236616 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:33.237888 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:33.237895 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:33.237919 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:23:33.237945 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:33.237987 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:23:33.238002 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:33.238201 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:33.238282 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 20:23:33.238386 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":348,"threads_started":1}
I20260620 20:23:33.239986 10491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:23:33.321976 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 10369
I20260620 20:23:33.338583 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 10018
I20260620 20:23:33.339851 10079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:23:33.446020 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 10018
2026-06-20T20:23:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2364 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:23:33 dist-test-slave-zzwf krb5kdc[10514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:23:33 dist-test-slave-zzwf krb5kdc[10514](info): set up 2 sockets
Jun 20 20:23:33 dist-test-slave-zzwf krb5kdc[10514](info): commencing operation
krb5kdc: starting...
W20260620 20:23:34.563148 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:23:34 dist-test-slave-zzwf krb5kdc[10514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:23:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:23:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:23:34.797809 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:45915
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:45915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:23:35.127172 10527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:35.127470 10527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:35.127560 10527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:35.134910 10527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:23:35.135030 10527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:35.135077 10527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:35.135126 10527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:23:35.135170 10527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:23:35.145040 10527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:45915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:45915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:35.146294 10527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:35.147518 10527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:35.159734 10533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:35.159945 10532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:35.159919 10535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:35.161242 10527 server_base.cc:1034] running on GCE node
I20260620 20:23:35.162129 10527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:35.163630 10527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:35.164887 10527 hybrid_clock.cc:648] HybridClock initialized: now 1781987015164831 us; error 47 us; skew 500 ppm
I20260620 20:23:35.165251 10527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:23:35 dist-test-slave-zzwf krb5kdc[10514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987015, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:23:35.168649 10527 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:23:35.170415 10527 webserver.cc:460] Webserver started at http://127.31.235.190:45621/ using document root <none> and password file <none>
I20260620 20:23:35.171151 10527 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:35.171311 10527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:35.171785 10527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:23:35.174930 10527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "4c1fa3d6dd5941c1b47b5cc406cbfa9b"
format_stamp: "Formatted at 2026-06-20 20:23:35 on dist-test-slave-zzwf"
server_key: "215b497629dd7808d3622a857ad74b51"
server_key_iv: "c4f8174544e25b0d42dafe86b33cc8a2"
server_key_version: "encryptionkey@0"
I20260620 20:23:35.175717 10527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "4c1fa3d6dd5941c1b47b5cc406cbfa9b"
format_stamp: "Formatted at 2026-06-20 20:23:35 on dist-test-slave-zzwf"
server_key: "215b497629dd7808d3622a857ad74b51"
server_key_iv: "c4f8174544e25b0d42dafe86b33cc8a2"
server_key_version: "encryptionkey@0"
I20260620 20:23:35.181011 10527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 20:23:35.184724 10542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:35.186328 10527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:23:35.186543 10527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "4c1fa3d6dd5941c1b47b5cc406cbfa9b"
format_stamp: "Formatted at 2026-06-20 20:23:35 on dist-test-slave-zzwf"
server_key: "215b497629dd7808d3622a857ad74b51"
server_key_iv: "c4f8174544e25b0d42dafe86b33cc8a2"
server_key_version: "encryptionkey@0"
I20260620 20:23:35.186755 10527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:35.205011 10527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:35.209317 10527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:35.209662 10527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:35.229000 10527 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:45915
I20260620 20:23:35.229041 10593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:45915 every 8 connection(s)
I20260620 20:23:35.230441 10527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:23:35.234179 10594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:35.240787 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 10527
I20260620 20:23:35.241042 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:23:35.241832 32686 external_mini_cluster.cc:1428] Setting key 0b71635c03f75222f94800af50fd617b
I20260620 20:23:35.245134 10594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b: Bootstrap starting.
I20260620 20:23:35.248575 10594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:35.249816 10594 log.cc:824] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:23:35.253096 10594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b: No bootstrap required, opened a new log
Jun 20 20:23:35 dist-test-slave-zzwf krb5kdc[10514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:23:35.263893 10594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1fa3d6dd5941c1b47b5cc406cbfa9b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 45915 } }
I20260620 20:23:35.264380 10594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:35.264494 10594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1fa3d6dd5941c1b47b5cc406cbfa9b, State: Initialized, Role: FOLLOWER
I20260620 20:23:35.265231 10594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1fa3d6dd5941c1b47b5cc406cbfa9b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 45915 } }
I20260620 20:23:35.265564 10594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:23:35.265697 10594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:23:35.265877 10594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:35.267254 10597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:35.243917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52270 (local address 127.31.235.190:45915)
0620 20:23:35.244687 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:35.244699 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:35.244740 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:23:35.246550 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:35.246574 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:35.246959 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:35.247281 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:35.249230 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:35.251025 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:35.252873 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:35.253712 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:35.253865 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:35.258431 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:35.258442 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:35.258468 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:35.258524 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:35.262640 (+ 4116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:35.263327 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:35.263337 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:35.263345 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:35.263538 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:35.263913 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:35.263921 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:35.263924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:35.264649 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:35.264966 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:35.265378 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 20:23:35.265858 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":383,"threads_started":1}
I20260620 20:23:35.268649 10594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1fa3d6dd5941c1b47b5cc406cbfa9b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 45915 } }
I20260620 20:23:35.269132 10594 leader_election.cc:302] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c1fa3d6dd5941c1b47b5cc406cbfa9b; no voters:
I20260620 20:23:35.269802 10594 leader_election.cc:288] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:23:35.269927 10599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:23:35.270681 10599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b [term 1 LEADER]: Becoming Leader. State: Replica: 4c1fa3d6dd5941c1b47b5cc406cbfa9b, State: Running, Role: LEADER
I20260620 20:23:35.271220 10599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1fa3d6dd5941c1b47b5cc406cbfa9b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 45915 } }
I20260620 20:23:35.272017 10594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:23:35.274091 10600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c1fa3d6dd5941c1b47b5cc406cbfa9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1fa3d6dd5941c1b47b5cc406cbfa9b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 45915 } } }
I20260620 20:23:35.274416 10600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b [sys.catalog]: This master's current role is: LEADER
I20260620 20:23:35.276086 10601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c1fa3d6dd5941c1b47b5cc406cbfa9b. Latest consensus state: current_term: 1 leader_uuid: "4c1fa3d6dd5941c1b47b5cc406cbfa9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c1fa3d6dd5941c1b47b5cc406cbfa9b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 45915 } } }
I20260620 20:23:35.276336 10605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:23:35.276355 10601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b [sys.catalog]: This master's current role is: LEADER
I20260620 20:23:35.282884 10605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:23:35.294457 10605 catalog_manager.cc:1347] Generated new cluster ID: 0e038182429a47148dc7a1fde09864b8
I20260620 20:23:35.294616 10605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:23:35.327010 10605 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:23:35.328379 10605 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:23:35.349310 10605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b: Generated new TSK 0
I20260620 20:23:35.350250 10605 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
I20260620 20:23:35.420778 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:45915
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:23:35.747009 10620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:35.747273 10620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:35.747364 10620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:35.754676 10620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:35.754788 10620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:35.754899 10620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:23:35.764596 10620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:45915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:35.765870 10620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:35.767143 10620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:35.780544 10625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:35.780570 10626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:35.780730 10628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:35.781746 10620 server_base.cc:1034] running on GCE node
I20260620 20:23:35.782356 10620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:35.783329 10620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:35.784533 10620 hybrid_clock.cc:648] HybridClock initialized: now 1781987015784472 us; error 62 us; skew 500 ppm
I20260620 20:23:35.784861 10620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:23:35 dist-test-slave-zzwf krb5kdc[10514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987015, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:23:35.788194 10620 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:23:35.789671 10620 webserver.cc:460] Webserver started at http://127.31.235.129:41213/ using document root <none> and password file <none>
I20260620 20:23:35.790346 10620 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:35.790477 10620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:35.790848 10620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:23:35.793884 10620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "fbccb798eb2248eb8b595392775661e5"
format_stamp: "Formatted at 2026-06-20 20:23:35 on dist-test-slave-zzwf"
server_key: "32ea4c3755375c1f4433779328d28fa2"
server_key_iv: "9dba3edc7a26ca6d5f5deb8e943dbea9"
server_key_version: "encryptionkey@0"
I20260620 20:23:35.794687 10620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "fbccb798eb2248eb8b595392775661e5"
format_stamp: "Formatted at 2026-06-20 20:23:35 on dist-test-slave-zzwf"
server_key: "32ea4c3755375c1f4433779328d28fa2"
server_key_iv: "9dba3edc7a26ca6d5f5deb8e943dbea9"
server_key_version: "encryptionkey@0"
I20260620 20:23:35.799583 10620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:23:35.803021 10635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:35.804425 10620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:23:35.804592 10620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "fbccb798eb2248eb8b595392775661e5"
format_stamp: "Formatted at 2026-06-20 20:23:35 on dist-test-slave-zzwf"
server_key: "32ea4c3755375c1f4433779328d28fa2"
server_key_iv: "9dba3edc7a26ca6d5f5deb8e943dbea9"
server_key_version: "encryptionkey@0"
I20260620 20:23:35.804791 10620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:35.823850 10620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:35.828073 10620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:35.828366 10620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:35.829275 10620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:23:35.830595 10620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:23:35.830699 10620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:35.830853 10620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:23:35.830935 10620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:35.863350 10620 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:33349
I20260620 20:23:35.863399 10747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:33349 every 8 connection(s)
I20260620 20:23:35.864629 10620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:23:35.866299 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 10620
I20260620 20:23:35.866518 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:23:35.867213 32686 external_mini_cluster.cc:1428] Setting key 18c0661d7f1d76356e195db902f8a588
Jun 20 20:23:35 dist-test-slave-zzwf krb5kdc[10514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987015, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:23:35.886549 10750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:35.867372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:36011 (local address 127.31.235.190:45915)
0620 20:23:35.867882 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:35.867887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:35.869900 (+ 2013us) server_negotiation.cc:407] Connection header received
0620 20:23:35.871630 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:35.871639 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:35.871746 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:35.871859 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:35.873450 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:35.874638 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:35.876164 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:35.876579 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:35.876723 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:35.880763 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:35.880768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:35.880771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:35.880802 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:35.883276 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:35.884190 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:35.884194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:35.884197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:35.884302 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:35.884791 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:35.884795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:35.884797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:35.885074 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:35.885190 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:35.886122 (+ 932us) server_negotiation.cc:299] Negotiation successful
0620 20:23:35.886289 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":242,"threads_started":1}
I20260620 20:23:35.888157 10751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:35.867967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:45915 (local address 127.31.235.129:36011)
0620 20:23:35.869734 (+ 1767us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:35.869785 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:35.871359 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:35.872056 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:35.872063 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:35.872689 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:35.873286 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:35.873314 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:35.874800 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:35.874804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:35.876004 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:35.876015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:35.876815 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:35.876824 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:35.877049 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:35.877762 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:35.877787 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:35.880550 (+ 2763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:23:35.883429 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:35.883441 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:35.883462 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:35.884078 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:35.884408 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:35.884415 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:35.884422 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:35.884656 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:35.885211 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:23:35.885223 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:23:35.885718 (+ 495us) client_negotiation.cc:769] Sending connection context
0620 20:23:35.886039 (+ 321us) client_negotiation.cc:240] Negotiation successful
0620 20:23:35.886629 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1469,"thread_start_us":1245,"threads_started":1}
I20260620 20:23:35.889858 10748 heartbeater.cc:347] Connected to a master server at 127.31.235.190:45915
I20260620 20:23:35.890158 10748 heartbeater.cc:464] Registering TS with master...
I20260620 20:23:35.890957 10748 heartbeater.cc:511] Master 127.31.235.190:45915 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.235.130@KRBTEST.COM; defaulting to no policy
I20260620 20:23:35.893181 10559 ts_manager.cc:194] Registered new tserver with Master: fbccb798eb2248eb8b595392775661e5 (127.31.235.129:33349)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.130@KRBTEST.COM" created.
I20260620 20:23:35.895249 10559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:36011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
I20260620 20:23:35.922734 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:45915
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:23:36.251277 10754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:36.251585 10754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:36.251731 10754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:36.260007 10754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:36.260171 10754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:36.260341 10754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:23:36.270270 10754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:45915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:36.271574 10754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:36.272975 10754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:36.286777 10759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:36.286756 10762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:36.287032 10760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:36.287827 10754 server_base.cc:1034] running on GCE node
I20260620 20:23:36.288473 10754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:36.289609 10754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:36.290851 10754 hybrid_clock.cc:648] HybridClock initialized: now 1781987016290785 us; error 66 us; skew 500 ppm
I20260620 20:23:36.291263 10754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:23:36 dist-test-slave-zzwf krb5kdc[10514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987016, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:23:36.294649 10754 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:23:36.296245 10754 webserver.cc:460] Webserver started at http://127.31.235.130:35587/ using document root <none> and password file <none>
I20260620 20:23:36.296983 10754 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:36.297134 10754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:36.297456 10754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:23:36.300590 10754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "c7c29e6382fe4273b84ac5396d3ee858"
format_stamp: "Formatted at 2026-06-20 20:23:36 on dist-test-slave-zzwf"
server_key: "e5166864cae5f46d7ce0060c9594bfdb"
server_key_iv: "b6e1ae3dc45d3607a0e4b2b776573343"
server_key_version: "encryptionkey@0"
I20260620 20:23:36.301450 10754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "c7c29e6382fe4273b84ac5396d3ee858"
format_stamp: "Formatted at 2026-06-20 20:23:36 on dist-test-slave-zzwf"
server_key: "e5166864cae5f46d7ce0060c9594bfdb"
server_key_iv: "b6e1ae3dc45d3607a0e4b2b776573343"
server_key_version: "encryptionkey@0"
I20260620 20:23:36.307116 10754 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:23:36.311043 10769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:36.312526 10754 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:23:36.312747 10754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "c7c29e6382fe4273b84ac5396d3ee858"
format_stamp: "Formatted at 2026-06-20 20:23:36 on dist-test-slave-zzwf"
server_key: "e5166864cae5f46d7ce0060c9594bfdb"
server_key_iv: "b6e1ae3dc45d3607a0e4b2b776573343"
server_key_version: "encryptionkey@0"
I20260620 20:23:36.312992 10754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:36.339885 10754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:36.343942 10754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:36.344257 10754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:36.345216 10754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:23:36.346540 10754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:23:36.346653 10754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:36.346807 10754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:23:36.346880 10754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:36.380400 10754 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:38897
I20260620 20:23:36.380543 10881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:38897 every 8 connection(s)
I20260620 20:23:36.381734 10754 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:23:36.384763 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 10754
I20260620 20:23:36.384954 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:23:36.385376 32686 external_mini_cluster.cc:1428] Setting key cf3c424ee0cfde4756ca2c26bfbe95f1
Jun 20 20:23:36 dist-test-slave-zzwf krb5kdc[10514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987016, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:23:36.403678 10750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:36.384240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:55543 (local address 127.31.235.190:45915)
0620 20:23:36.384418 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:36.384422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:36.385772 (+ 1350us) server_negotiation.cc:407] Connection header received
0620 20:23:36.387795 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:36.387801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:36.387881 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:36.388012 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:36.389579 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.390696 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.392606 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.393355 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.393523 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:36.396886 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:36.396891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:36.396894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:36.396923 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:36.400748 (+ 3825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:36.401444 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:36.401450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:36.401453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:36.401625 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:36.402042 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:36.402047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:36.402050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:36.402309 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:36.402468 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:36.403124 (+ 656us) server_negotiation.cc:299] Negotiation successful
0620 20:23:36.403339 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:23:36.404744 10884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:36.384715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:45915 (local address 127.31.235.130:55543)
0620 20:23:36.385594 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:36.385639 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:36.387534 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:36.388190 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:36.388197 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:36.388712 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:36.389395 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:36.389414 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.390879 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.390906 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:36.392458 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:36.392475 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.393602 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.393608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:36.393791 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:36.394486 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:36.394508 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:36.396662 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:23:36.400885 (+ 4223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:36.400895 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:36.400911 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:36.401315 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:36.401731 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:36.401737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:36.401740 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:36.401923 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:36.402440 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:23:36.402450 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:23:36.402768 (+ 318us) client_negotiation.cc:769] Sending connection context
0620 20:23:36.403030 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 20:23:36.403551 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":379,"threads_started":1}
I20260620 20:23:36.406303 10882 heartbeater.cc:347] Connected to a master server at 127.31.235.190:45915
I20260620 20:23:36.406605 10882 heartbeater.cc:464] Registering TS with master...
I20260620 20:23:36.407286 10882 heartbeater.cc:511] Master 127.31.235.190:45915 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.235.131@KRBTEST.COM; defaulting to no policy
I20260620 20:23:36.408748 10559 ts_manager.cc:194] Registered new tserver with Master: c7c29e6382fe4273b84ac5396d3ee858 (127.31.235.130:38897)
I20260620 20:23:36.409641 10559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:55543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
I20260620 20:23:36.436586 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:45915
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:23:36.756857 10887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:23:36.757153 10887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:23:36.757341 10887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:23:36.764451 10887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:23:36.764593 10887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:23:36.764783 10887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:23:36.774493 10887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:45915
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:23:36.775879 10887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:23:36.777235 10887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:23:36.790550 10893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:36.790731 10892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:23:36.791052 10895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:23:36.791715 10887 server_base.cc:1034] running on GCE node
I20260620 20:23:36.792343 10887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:23:36.793376 10887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:23:36.794608 10887 hybrid_clock.cc:648] HybridClock initialized: now 1781987016794558 us; error 48 us; skew 500 ppm
I20260620 20:23:36.794934 10887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:23:36 dist-test-slave-zzwf krb5kdc[10514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987016, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:23:36.798187 10887 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:23:36.799559 10887 webserver.cc:460] Webserver started at http://127.31.235.131:45655/ using document root <none> and password file <none>
I20260620 20:23:36.800203 10887 fs_manager.cc:359] Metadata directory not provided
I20260620 20:23:36.800303 10887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:23:36.800552 10887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:23:36.803419 10887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "9e90151a953a4365805d4c033be66944"
format_stamp: "Formatted at 2026-06-20 20:23:36 on dist-test-slave-zzwf"
server_key: "f20b9a616cbc3bb0724aa00c736a3f0d"
server_key_iv: "8d422210eb5709e6a3adbaf7efecce5b"
server_key_version: "encryptionkey@0"
I20260620 20:23:36.804128 10887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "9e90151a953a4365805d4c033be66944"
format_stamp: "Formatted at 2026-06-20 20:23:36 on dist-test-slave-zzwf"
server_key: "f20b9a616cbc3bb0724aa00c736a3f0d"
server_key_iv: "8d422210eb5709e6a3adbaf7efecce5b"
server_key_version: "encryptionkey@0"
I20260620 20:23:36.809193 10887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 20:23:36.812438 10902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:36.813840 10887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:23:36.813969 10887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "9e90151a953a4365805d4c033be66944"
format_stamp: "Formatted at 2026-06-20 20:23:36 on dist-test-slave-zzwf"
server_key: "f20b9a616cbc3bb0724aa00c736a3f0d"
server_key_iv: "8d422210eb5709e6a3adbaf7efecce5b"
server_key_version: "encryptionkey@0"
I20260620 20:23:36.814147 10887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:23:36.833627 10887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:23:36.837795 10887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:23:36.838052 10887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:23:36.838953 10887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:23:36.840246 10887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:23:36.840328 10887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:36.840426 10887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:23:36.840476 10887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:23:36.875490 10887 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:42299
I20260620 20:23:36.875517 11014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:42299 every 8 connection(s)
I20260620 20:23:36.876803 10887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:23:36.878016 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 10887
I20260620 20:23:36.878204 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:23:36.878700 32686 external_mini_cluster.cc:1428] Setting key d821b04b4696119a58608a2659401527
Jun 20 20:23:36 dist-test-slave-zzwf krb5kdc[10514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987016, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:23:36.896816 10750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:36.879741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:51755 (local address 127.31.235.190:45915)
0620 20:23:36.879911 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:36.879916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:36.881366 (+ 1450us) server_negotiation.cc:407] Connection header received
0620 20:23:36.883308 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:36.883314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:36.883408 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:36.883491 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:36.884876 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.885685 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.887106 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.887507 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.887606 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:36.891086 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:36.891091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:36.891094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:36.891124 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:36.894153 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:36.894808 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:36.894812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:36.894814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:36.894928 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:36.895323 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:36.895329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:36.895332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:36.895636 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:36.895771 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:36.896402 (+ 631us) server_negotiation.cc:299] Negotiation successful
0620 20:23:36.896564 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 20:23:36.897836 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:36.880294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:45915 (local address 127.31.235.131:51755)
0620 20:23:36.881193 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:36.881240 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:36.883026 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:36.883691 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:36.883699 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:36.884175 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:23:36.884725 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:36.884742 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.885811 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.885815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:36.886991 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:36.887001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.887727 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.887732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:36.887931 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:36.888642 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:23:36.888669 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:23:36.890901 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:23:36.894276 (+ 3375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:36.894282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:36.894295 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:36.894713 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:36.895030 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:23:36.895033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:23:36.895036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:23:36.895194 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:23:36.895755 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:23:36.895761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:23:36.896056 (+ 295us) client_negotiation.cc:769] Sending connection context
0620 20:23:36.896284 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 20:23:36.896653 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":387,"threads_started":1}
I20260620 20:23:36.898990 10748 heartbeater.cc:503] Master 127.31.235.190:45915 was elected leader, sending a full tablet report...
I20260620 20:23:36.899391 11015 heartbeater.cc:347] Connected to a master server at 127.31.235.190:45915
I20260620 20:23:36.899744 11015 heartbeater.cc:464] Registering TS with master...
I20260620 20:23:36.900592 11015 heartbeater.cc:511] Master 127.31.235.190:45915 requested a full tablet report, sending...
I20260620 20:23:36.901947 10559 ts_manager.cc:194] Registered new tserver with Master: 9e90151a953a4365805d4c033be66944 (127.31.235.131:42299)
I20260620 20:23:36.902647 10559 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:51755
I20260620 20:23:36.906262 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:23:36.922708 10750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:36.910705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55848 (local address 127.31.235.190:45915)
0620 20:23:36.910884 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:36.910889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:36.911405 (+ 516us) server_negotiation.cc:407] Connection header received
0620 20:23:36.911510 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:36.911514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:36.911571 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:36.911653 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:36.912940 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.913773 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.914992 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.915386 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.915519 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:36.917727 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:36.917732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:36.917735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:36.917763 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:36.920396 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:36.921029 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:36.921033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:36.921035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:36.921154 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:36.921556 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:36.921560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:36.921562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:36.921838 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:36.922027 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:36.922310 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 20:23:36.922488 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:23:36.926647 10559 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:23:36.933241 10559 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:23:36.958796 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:36.945274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:42299 (local address 127.0.0.1:46810)
0620 20:23:36.946650 (+ 1376us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:36.946681 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:36.946896 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:36.948105 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:36.948117 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:36.948162 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:23:36.948725 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:36.948744 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.951826 (+ 3082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.951836 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:36.955503 (+ 3667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:36.955529 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.957654 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.957662 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:36.957939 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:36.958015 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 20:23:36.958146 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 20:23:36.958358 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1189,"threads_started":1}
I20260620 20:23:36.960052 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:36.945461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46810 (local address 127.31.235.131:42299)
0620 20:23:36.946310 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:36.946321 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:36.946793 (+ 472us) server_negotiation.cc:407] Connection header received
0620 20:23:36.947118 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:36.947131 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:36.947449 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:36.947687 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:23:36.949035 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.951583 (+ 2548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.955745 (+ 4162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.957374 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.959225 (+ 1851us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:36.959445 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:36.959518 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 20:23:36.959684 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":465,"threads_started":1}
I20260620 20:23:36.965029 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:36.945348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:33349 (local address 127.0.0.1:44376)
0620 20:23:36.956281 (+ 10933us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:36.956302 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:36.956433 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:36.957224 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:36.957229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:36.957258 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:23:36.957806 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:36.957819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.960124 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.960133 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:36.962088 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:36.962101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.964196 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.964203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:36.964374 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:36.964410 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 20:23:36.964493 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 20:23:36.964648 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10788,"thread_start_us":944,"threads_started":1}
I20260620 20:23:36.964143 10950 tablet_service.cc:1467] Processing CreateTablet for tablet 37b0adced4e44e8992f5888c19f4d71d (DEFAULT_TABLE table=test-table [id=1992a811ff144bb1a36021f678802aaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:23:36.965453 11030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:36.945877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44376 (local address 127.31.235.129:33349)
0620 20:23:36.956004 (+ 10127us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:36.956012 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:36.956572 (+ 560us) server_negotiation.cc:407] Connection header received
0620 20:23:36.956679 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:36.956688 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:36.956991 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:36.958051 (+ 1060us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:23:36.958305 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.959937 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.962311 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.963931 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.964724 (+ 793us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:36.964884 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:36.964987 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 20:23:36.965147 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9907,"thread_start_us":9720,"threads_started":1}
I20260620 20:23:36.965945 10950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b0adced4e44e8992f5888c19f4d71d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:36.967330 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:36.947737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.31.235.130:38897)
0620 20:23:36.959270 (+ 11533us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:36.959277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:36.959300 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:23:36.959369 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:36.959374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:36.959547 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:36.960371 (+ 824us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:23:36.960594 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.962056 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.964820 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.966394 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.966598 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:36.966737 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:36.966993 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 20:23:36.967116 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11390,"thread_start_us":11298,"threads_started":1}
I20260620 20:23:36.967409 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:36.947230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:38897 (local address 127.0.0.1:52740)
0620 20:23:36.958401 (+ 11171us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:36.958425 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:36.958579 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:36.959725 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:36.959729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:36.959754 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:23:36.960079 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:36.960094 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.962202 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.962211 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:36.964634 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:36.964647 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:36.966628 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:36.966635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:36.966819 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:36.966850 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 20:23:36.966953 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 20:23:36.967084 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10981,"thread_start_us":8759,"threads_started":1}
I20260620 20:23:36.969307 10683 tablet_service.cc:1467] Processing CreateTablet for tablet 37b0adced4e44e8992f5888c19f4d71d (DEFAULT_TABLE table=test-table [id=1992a811ff144bb1a36021f678802aaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:23:36.970754 10683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b0adced4e44e8992f5888c19f4d71d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:36.971132 10817 tablet_service.cc:1467] Processing CreateTablet for tablet 37b0adced4e44e8992f5888c19f4d71d (DEFAULT_TABLE table=test-table [id=1992a811ff144bb1a36021f678802aaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:23:36.972354 10817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b0adced4e44e8992f5888c19f4d71d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:23:36.985737 11033 tablet_bootstrap.cc:492] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944: Bootstrap starting.
I20260620 20:23:36.986793 11034 tablet_bootstrap.cc:492] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5: Bootstrap starting.
I20260620 20:23:36.989071 11035 tablet_bootstrap.cc:492] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858: Bootstrap starting.
I20260620 20:23:36.990329 11033 tablet_bootstrap.cc:654] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:36.990339 11034 tablet_bootstrap.cc:654] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:36.991515 11033 log.cc:824] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944: Log is configured to *not* fsync() on all Append() calls
I20260620 20:23:36.991576 11034 log.cc:824] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5: Log is configured to *not* fsync() on all Append() calls
I20260620 20:23:36.992600 11035 tablet_bootstrap.cc:654] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858: Neither blocks nor log segments found. Creating new log.
I20260620 20:23:36.993866 11035 log.cc:824] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858: Log is configured to *not* fsync() on all Append() calls
I20260620 20:23:36.994746 11034 tablet_bootstrap.cc:492] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5: No bootstrap required, opened a new log
I20260620 20:23:36.994772 11033 tablet_bootstrap.cc:492] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944: No bootstrap required, opened a new log
I20260620 20:23:36.995041 11033 ts_tablet_manager.cc:1397] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260620 20:23:36.995049 11034 ts_tablet_manager.cc:1397] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260620 20:23:36.996928 11035 tablet_bootstrap.cc:492] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858: No bootstrap required, opened a new log
I20260620 20:23:36.997180 11035 ts_tablet_manager.cc:1397] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 20:23:37.002538 11033 raft_consensus.cc:348] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e90151a953a4365805d4c033be66944" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42299 } } peers { permanent_uuid: "fbccb798eb2248eb8b595392775661e5" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 33349 } } peers { permanent_uuid: "c7c29e6382fe4273b84ac5396d3ee858" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38897 } }
I20260620 20:23:37.002929 11033 raft_consensus.cc:374] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:37.003036 11033 raft_consensus.cc:729] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e90151a953a4365805d4c033be66944, State: Initialized, Role: FOLLOWER
I20260620 20:23:37.003669 11033 consensus_queue.cc:260] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e90151a953a4365805d4c033be66944" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42299 } } peers { permanent_uuid: "fbccb798eb2248eb8b595392775661e5" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 33349 } } peers { permanent_uuid: "c7c29e6382fe4273b84ac5396d3ee858" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38897 } }
I20260620 20:23:37.004776 11015 heartbeater.cc:503] Master 127.31.235.190:45915 was elected leader, sending a full tablet report...
I20260620 20:23:37.005462 11033 ts_tablet_manager.cc:1428] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260620 20:23:37.005725 11034 raft_consensus.cc:348] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e90151a953a4365805d4c033be66944" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42299 } } peers { permanent_uuid: "fbccb798eb2248eb8b595392775661e5" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 33349 } } peers { permanent_uuid: "c7c29e6382fe4273b84ac5396d3ee858" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38897 } }
I20260620 20:23:37.006172 11034 raft_consensus.cc:374] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:37.006271 11034 raft_consensus.cc:729] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbccb798eb2248eb8b595392775661e5, State: Initialized, Role: FOLLOWER
I20260620 20:23:37.006979 11034 consensus_queue.cc:260] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e90151a953a4365805d4c033be66944" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42299 } } peers { permanent_uuid: "fbccb798eb2248eb8b595392775661e5" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 33349 } } peers { permanent_uuid: "c7c29e6382fe4273b84ac5396d3ee858" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38897 } }
I20260620 20:23:37.008234 11035 raft_consensus.cc:348] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e90151a953a4365805d4c033be66944" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42299 } } peers { permanent_uuid: "fbccb798eb2248eb8b595392775661e5" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 33349 } } peers { permanent_uuid: "c7c29e6382fe4273b84ac5396d3ee858" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38897 } }
I20260620 20:23:37.008684 11035 raft_consensus.cc:374] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:23:37.008780 11035 raft_consensus.cc:729] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7c29e6382fe4273b84ac5396d3ee858, State: Initialized, Role: FOLLOWER
I20260620 20:23:37.008812 11034 ts_tablet_manager.cc:1428] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260620 20:23:37.009370 11035 consensus_queue.cc:260] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e90151a953a4365805d4c033be66944" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42299 } } peers { permanent_uuid: "fbccb798eb2248eb8b595392775661e5" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 33349 } } peers { permanent_uuid: "c7c29e6382fe4273b84ac5396d3ee858" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38897 } }
I20260620 20:23:37.010507 10882 heartbeater.cc:503] Master 127.31.235.190:45915 was elected leader, sending a full tablet report...
I20260620 20:23:37.011096 11035 ts_tablet_manager.cc:1428] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
W20260620 20:23:37.116977 10749 tablet.cc:2369] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:23:37.128523 11016 tablet.cc:2369] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:23:37.133703 10883 tablet.cc:2369] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:23:37.299841 11040 raft_consensus.cc:482] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:23:37.300166 11040 raft_consensus.cc:504] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e90151a953a4365805d4c033be66944" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42299 } } peers { permanent_uuid: "fbccb798eb2248eb8b595392775661e5" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 33349 } } peers { permanent_uuid: "c7c29e6382fe4273b84ac5396d3ee858" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38897 } }
I20260620 20:23:37.301873 11040 leader_election.cc:288] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e90151a953a4365805d4c033be66944 (127.31.235.131:42299), c7c29e6382fe4273b84ac5396d3ee858 (127.31.235.130:38897)
I20260620 20:23:37.308828 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:37.302414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:52515 (local address 127.31.235.130:38897)
0620 20:23:37.302621 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:37.302628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:37.303282 (+ 654us) server_negotiation.cc:407] Connection header received
0620 20:23:37.303418 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:37.303423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:37.303514 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:37.303606 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:23:37.304239 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:37.304995 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:37.307016 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:37.307809 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:37.307941 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:37.308019 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:37.308403 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 20:23:37.308561 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 20:23:37.308925 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:37.302276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:49995 (local address 127.31.235.131:42299)
0620 20:23:37.302476 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:37.302485 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:37.303421 (+ 936us) server_negotiation.cc:407] Connection header received
0620 20:23:37.303604 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:37.303609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:37.303680 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:37.303780 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:23:37.304346 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:37.305098 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:37.306956 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:37.307929 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:37.308098 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:37.308188 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:37.308554 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 20:23:37.308696 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:23:37.309054 11042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:37.302330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:38897 (local address 127.31.235.129:52515)
0620 20:23:37.303166 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:37.303182 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:37.303303 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:37.303633 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:37.303637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:37.303660 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:23:37.304083 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:37.304092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:37.305120 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:37.305128 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:37.306845 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:37.306856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:37.308017 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:37.308026 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:37.308241 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:37.308278 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 20:23:37.308352 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 20:23:37.308502 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":114,"thread_start_us":497,"threads_started":1}
I20260620 20:23:37.309123 11043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:37.302406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:42299 (local address 127.31.235.129:49995)
0620 20:23:37.303287 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0620 20:23:37.303305 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:23:37.303454 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:23:37.303805 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:23:37.303808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:23:37.303826 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:23:37.304172 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:37.304185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:37.305236 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:37.305246 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:37.306806 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:23:37.306817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:23:37.308143 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:23:37.308147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:23:37.308344 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:37.308381 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 20:23:37.308512 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 20:23:37.308642 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":9,"thread_start_us":509,"threads_started":1}
I20260620 20:23:37.310012 10970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b0adced4e44e8992f5888c19f4d71d" candidate_uuid: "fbccb798eb2248eb8b595392775661e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e90151a953a4365805d4c033be66944" is_pre_election: true
I20260620 20:23:37.310062 10837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b0adced4e44e8992f5888c19f4d71d" candidate_uuid: "fbccb798eb2248eb8b595392775661e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7c29e6382fe4273b84ac5396d3ee858" is_pre_election: true
I20260620 20:23:37.310544 10837 raft_consensus.cc:2454] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbccb798eb2248eb8b595392775661e5 in term 0.
I20260620 20:23:37.310532 10970 raft_consensus.cc:2454] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbccb798eb2248eb8b595392775661e5 in term 0.
I20260620 20:23:37.311203 10639 leader_election.cc:302] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e90151a953a4365805d4c033be66944, fbccb798eb2248eb8b595392775661e5; no voters:
I20260620 20:23:37.311578 11040 raft_consensus.cc:2781] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:23:37.311755 11040 raft_consensus.cc:482] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:23:37.311878 11040 raft_consensus.cc:3037] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:37.314137 11040 raft_consensus.cc:504] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e90151a953a4365805d4c033be66944" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42299 } } peers { permanent_uuid: "fbccb798eb2248eb8b595392775661e5" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 33349 } } peers { permanent_uuid: "c7c29e6382fe4273b84ac5396d3ee858" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38897 } }
I20260620 20:23:37.314669 11040 leader_election.cc:288] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 [CANDIDATE]: Term 1 election: Requested vote from peers 9e90151a953a4365805d4c033be66944 (127.31.235.131:42299), c7c29e6382fe4273b84ac5396d3ee858 (127.31.235.130:38897)
I20260620 20:23:37.315251 10970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b0adced4e44e8992f5888c19f4d71d" candidate_uuid: "fbccb798eb2248eb8b595392775661e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e90151a953a4365805d4c033be66944"
I20260620 20:23:37.315251 10837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b0adced4e44e8992f5888c19f4d71d" candidate_uuid: "fbccb798eb2248eb8b595392775661e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7c29e6382fe4273b84ac5396d3ee858"
I20260620 20:23:37.315480 10970 raft_consensus.cc:3037] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:37.315470 10837 raft_consensus.cc:3037] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:23:37.318735 10970 raft_consensus.cc:2454] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbccb798eb2248eb8b595392775661e5 in term 1.
I20260620 20:23:37.318859 10837 raft_consensus.cc:2454] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbccb798eb2248eb8b595392775661e5 in term 1.
I20260620 20:23:37.319273 10638 leader_election.cc:302] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9e90151a953a4365805d4c033be66944, c7c29e6382fe4273b84ac5396d3ee858, fbccb798eb2248eb8b595392775661e5; no voters:
I20260620 20:23:37.319650 11040 raft_consensus.cc:2781] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:23:37.320111 11040 raft_consensus.cc:686] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 [term 1 LEADER]: Becoming Leader. State: Replica: fbccb798eb2248eb8b595392775661e5, State: Running, Role: LEADER
I20260620 20:23:37.320613 11040 consensus_queue.cc:237] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e90151a953a4365805d4c033be66944" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42299 } } peers { permanent_uuid: "fbccb798eb2248eb8b595392775661e5" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 33349 } } peers { permanent_uuid: "c7c29e6382fe4273b84ac5396d3ee858" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38897 } }
I20260620 20:23:37.325735 10557 catalog_manager.cc:5495] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fbccb798eb2248eb8b595392775661e5 (127.31.235.129). New cstate: current_term: 1 leader_uuid: "fbccb798eb2248eb8b595392775661e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e90151a953a4365805d4c033be66944" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbccb798eb2248eb8b595392775661e5" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 33349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7c29e6382fe4273b84ac5396d3ee858" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38897 } health_report { overall_health: UNKNOWN } } }
I20260620 20:23:37.423452 11030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:37.417739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.31.235.129:33349)
0620 20:23:37.418014 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:37.418018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:37.418103 (+ 85us) server_negotiation.cc:407] Connection header received
0620 20:23:37.418307 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:37.418312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:37.418419 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:37.418600 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:23:37.419067 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:37.419818 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:37.421112 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:37.421624 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:37.421739 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:37.422277 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:23:37.422830 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:23:37.422949 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:37.423132 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 20:23:37.423213 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260620 20:23:37.431370 10837 raft_consensus.cc:1261] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858 [term 1 FOLLOWER]: Refusing update from remote peer fbccb798eb2248eb8b595392775661e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:23:37.431411 10970 raft_consensus.cc:1261] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944 [term 1 FOLLOWER]: Refusing update from remote peer fbccb798eb2248eb8b595392775661e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:23:37.432325 11044 consensus_queue.cc:1035] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7c29e6382fe4273b84ac5396d3ee858" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:23:37.432649 11040 consensus_queue.cc:1035] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e90151a953a4365805d4c033be66944" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:23:37.456472 10557 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:23:37.456770 10557 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:23:37.460245 10557 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c1fa3d6dd5941c1b47b5cc406cbfa9b: Sending DeleteTablet for 3 replicas of tablet 37b0adced4e44e8992f5888c19f4d71d
I20260620 20:23:37.461380 10950 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b0adced4e44e8992f5888c19f4d71d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:23:37 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:46810
I20260620 20:23:37.461407 10683 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b0adced4e44e8992f5888c19f4d71d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:23:37 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:44376
I20260620 20:23:37.461452 10817 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b0adced4e44e8992f5888c19f4d71d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:23:37 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:52740
I20260620 20:23:37.463639 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 10620
I20260620 20:23:37.470069 11059 tablet_replica.cc:330] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858: stopping tablet replica
I20260620 20:23:37.470849 11057 tablet_replica.cc:330] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944: stopping tablet replica
I20260620 20:23:37.471303 11059 raft_consensus.cc:2229] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:23:37.471303 11057 raft_consensus.cc:2229] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:23:37.471823 11057 raft_consensus.cc:2258] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:23:37.472332 11059 raft_consensus.cc:2258] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:23:37.474264 11057 ts_tablet_manager.cc:1905] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:23:37.478530 11058 tablet_replica.cc:330] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5: stopping tablet replica
I20260620 20:23:37.478783 11059 ts_tablet_manager.cc:1905] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:23:37.478950 11058 raft_consensus.cc:2229] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:23:37.479630 11058 raft_consensus.cc:2258] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:23:37.481683 11058 ts_tablet_manager.cc:1905] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:23:37.483095 11057 ts_tablet_manager.cc:1918] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:23:37.483249 11057 log.cc:1196] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/37b0adced4e44e8992f5888c19f4d71d
I20260620 20:23:37.483731 11057 ts_tablet_manager.cc:1939] T 37b0adced4e44e8992f5888c19f4d71d P 9e90151a953a4365805d4c033be66944: Deleting consensus metadata
I20260620 20:23:37.485584 11059 ts_tablet_manager.cc:1918] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:23:37.485546 10546 catalog_manager.cc:4841] TS 9e90151a953a4365805d4c033be66944 (127.31.235.131:42299): tablet 37b0adced4e44e8992f5888c19f4d71d (table test-table [id=1992a811ff144bb1a36021f678802aaf]) successfully deleted
I20260620 20:23:37.485731 11059 log.cc:1196] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/37b0adced4e44e8992f5888c19f4d71d
I20260620 20:23:37.486235 11059 ts_tablet_manager.cc:1939] T 37b0adced4e44e8992f5888c19f4d71d P c7c29e6382fe4273b84ac5396d3ee858: Deleting consensus metadata
I20260620 20:23:37.487897 10545 catalog_manager.cc:4841] TS c7c29e6382fe4273b84ac5396d3ee858 (127.31.235.130:38897): tablet 37b0adced4e44e8992f5888c19f4d71d (table test-table [id=1992a811ff144bb1a36021f678802aaf]) successfully deleted
Jun 20 20:23:37 dist-test-slave-zzwf krb5kdc[10514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:23:37.490967 11058 ts_tablet_manager.cc:1918] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:23:37.491132 11058 log.cc:1196] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/37b0adced4e44e8992f5888c19f4d71d
I20260620 20:23:37.491629 11058 ts_tablet_manager.cc:1939] T 37b0adced4e44e8992f5888c19f4d71d P fbccb798eb2248eb8b595392775661e5: Deleting consensus metadata
I20260620 20:23:37.493184 10545 catalog_manager.cc:4841] TS fbccb798eb2248eb8b595392775661e5 (127.31.235.129:33349): tablet 37b0adced4e44e8992f5888c19f4d71d (table test-table [id=1992a811ff144bb1a36021f678802aaf]) successfully deleted
I20260620 20:23:37.495635 11030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:37.467932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44392 (local address 127.31.235.129:33349)
0620 20:23:37.468160 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:37.468169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:37.470239 (+ 2070us) server_negotiation.cc:407] Connection header received
0620 20:23:37.470440 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:37.470450 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:37.470584 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:37.470712 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:37.473051 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:37.474661 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:37.477139 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:37.478163 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:37.478288 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:37.489223 (+ 10935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:37.489238 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:37.489252 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:37.489293 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:37.492753 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:37.493622 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:37.493633 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:37.493641 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:37.493765 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:37.494153 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:37.494157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:37.494159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:37.494475 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:37.494771 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:37.495082 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 20:23:37.495318 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:23:37.496299 10743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:23:37.592633 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 10620
I20260620 20:23:37.607357 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 10754
Jun 20 20:23:37 dist-test-slave-zzwf krb5kdc[10514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.130@KRBTEST.COM
I20260620 20:23:37.623242 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:37.608214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52746 (local address 127.31.235.130:38897)
0620 20:23:37.608422 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:37.608427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:37.608447 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:23:37.608562 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:37.608566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:37.608648 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:37.608753 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:37.610004 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:37.610744 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:37.611945 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:37.612413 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:37.612525 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:37.616909 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:37.616925 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:37.616941 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:37.617012 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:37.620272 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:37.621022 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:37.621031 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:37.621033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:37.621191 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:37.621647 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:37.621653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:37.621658 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:37.622050 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:37.622363 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:37.622656 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 20:23:37.622890 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:23:37.623934 10877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:23:37.712405 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 10754
I20260620 20:23:37.729280 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 10887
Jun 20 20:23:37 dist-test-slave-zzwf krb5kdc[10514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.131@KRBTEST.COM
I20260620 20:23:37.743391 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:23:37.730074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.31.235.131:42299)
0620 20:23:37.730246 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 20:23:37.730250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:23:37.730270 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:23:37.730349 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:23:37.730353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:23:37.730415 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:23:37.730510 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:23:37.731766 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:37.732469 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:37.733788 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:23:37.734172 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:23:37.734264 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:23:37.738361 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:23:37.738376 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:23:37.738393 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:23:37.738428 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:23:37.740896 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:37.741607 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:37.741614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:37.741616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:37.741751 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:23:37.742067 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:23:37.742070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:23:37.742072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:23:37.742389 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:23:37.742592 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 20:23:37.742861 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 20:23:37.743018 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:23:37.744127 11010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:23:37.834924 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 10887
I20260620 20:23:37.851679 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 10527
I20260620 20:23:37.852824 10589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:23:37.960016 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 10527
2026-06-20T20:23:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4517 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:23:38 dist-test-slave-zzwf krb5kdc[11067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:23:38 dist-test-slave-zzwf krb5kdc[11067](info): set up 2 sockets
Jun 20 20:23:38 dist-test-slave-zzwf krb5kdc[11067](info): commencing operation
krb5kdc: starting...
W20260620 20:23:39.118960 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:23:39 dist-test-slave-zzwf krb5kdc[11067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:23:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:23:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.235.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.235.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.235.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/rangeradmin_127.31.235.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.235.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/rangeradmin_127.31.235.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.235.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.235.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.235.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/rangerlookup_127.31.235.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.235.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/rangerlookup_127.31.235.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.235.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.235.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.235.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/HTTP_127.31.235.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.235.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/HTTP_127.31.235.148@KRBTEST.COM.keytab.
I20260620 20:23:39.456995 32686 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskgJsYo5/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/postgres -l logfile start
2026-06-20 20:23:42.216 UTC [11097] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 20:23:42.216 UTC [11097] LOG: listening on IPv4 address "127.31.235.148", port 48935
2026-06-20 20:23:42.220 UTC [11097] LOG: listening on Unix socket "/tmp/.s.PGSQL.48935"
2026-06-20 20:23:42.243 UTC [11100] LOG: database system was shut down at 2026-06-20 20:23:40 UTC
2026-06-20 20:23:42.247 UTC [11097] LOG: database system is ready to accept connections
I20260620 20:23:43.243080 32686 mini_postgres.cc:96] Postgres bound to 48935
2026-06-20 20:23:43.261 UTC [11108] FATAL: database "slave" does not exist
127.31.235.148:48935 - accepting connections
I20260620 20:23:43.262004 32686 mini_ranger.cc:161] Starting Ranger...
I20260620 20:23:43.291397 32686 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 20:23:43.875407 32686 mini_ranger.cc:87] Created ranger Postgres database
I20260620 20:23:43.875514 32686 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ranger-admin
2026-06-20 20:23:44,181 [I] DB FLAVOR :POSTGRES
2026-06-20 20:23:44,181 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 20:23:44,181 [I] Checking connection..
2026-06-20 20:23:44,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 20:23:44,457 [I] Checking connection passed.
2026-06-20 20:23:44,457 [I] --------- Verifying version history table ---------
2026-06-20 20:23:44,457 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:23:44,721 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 20:23:44,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:23:45,000 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 20:23:45,000 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 20:23:45,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 20:23:45.268 UTC [11172] WARNING: there is no transaction in progress
2026-06-20 20:23:45,275 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 20:23:45,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:23:45,527 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 20:23:45,528 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 20:23:45,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 20:23:45,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 20:23:46,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zzwf.c.gcp-upstream.internal','N') ;"
2026-06-20 20:23:46,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 20:23:46,542 [I] Table x_portal_user does not exist in database ranger
2026-06-20 20:23:46,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 20:23:46,782 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 20:23:46,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 20:23:47,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 20:23:47,268 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 20:23:47,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 20:23:48.349 UTC [11280] WARNING: there is no transaction in progress
2026-06-20 20:23:48.366 UTC [11280] WARNING: there is no transaction in progress
2026-06-20 20:23:48.383 UTC [11280] WARNING: there is no transaction in progress
2026-06-20 20:23:48.400 UTC [11280] WARNING: there is no transaction in progress
2026-06-20 20:23:48.418 UTC [11280] WARNING: there is no transaction in progress
2026-06-20 20:23:48.436 UTC [11280] WARNING: there is no transaction in progress
2026-06-20 20:23:48.497 UTC [11280] WARNING: there is no transaction in progress
2026-06-20 20:23:48.505 UTC [11280] WARNING: there is no transaction in progress
2026-06-20 20:23:48.524 UTC [11280] WARNING: there is no transaction in progress
2026-06-20 20:23:48.537 UTC [11280] WARNING: there is no transaction in progress
2026-06-20 20:23:48.550 UTC [11280] WARNING: there is no transaction in progress
2026-06-20 20:23:48.566 UTC [11280] WARNING: there is no transaction in progress
2026-06-20 20:23:48.577 UTC [11280] WARNING: there is no transaction in progress
2026-06-20 20:23:48.583 UTC [11280] WARNING: there is no transaction in progress
2026-06-20 20:23:48.591 UTC [11280] WARNING: there is no transaction in progress
2026-06-20 20:23:48.598 UTC [11280] WARNING: there is no transaction in progress
2026-06-20 20:23:49,361 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 20:23:49,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 20:23:49,604 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 20:23:49,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 20:23:49,837 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 20:23:49,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 20:23:50,102 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 20:23:50,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 20:23:50,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 20:23:50,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zzwf.c.gcp-upstream.internal';"
2026-06-20 20:23:50,830 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 20:23:50,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 20:23:51,089 [I] DB_PATCHES have already been applied
I20260620 20:23:51.095110 32686 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ranger-admin:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskgJsYo5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 20:23:51.095199 32686 mini_ranger.cc:193] Using host: 127.31.235.148
I20260620 20:23:51.110373 32686 mini_ranger.cc:227] Ranger admin URL: http://127.31.235.148:50317
Jun 20, 2026 8:23:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:23:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:23:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 8:23:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 8:23:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 8:23:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 8:23:52 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 20:23:52.265628 32686 mini_ranger.cc:160] Time spent starting Ranger: real 9.004s user 0.005s sys 0.036s
Jun 20, 2026 8:24:11 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 8:24:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 8:24:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 8:24:11 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 8:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 8:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 8:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 8:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 8:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 8:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 8:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 8:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 8:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 8:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 8:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 8:24:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 8:24:11 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 8:24:12 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 20:24:12.932545 32686 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.235.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.235.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.235.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/rangerkms_127.31.235.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.235.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/rangerkms_127.31.235.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.235.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.235.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.235.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/HTTP_127.31.235.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.235.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/HTTP_127.31.235.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 20:24:13 dist-test-slave-zzwf krb5kdc[11067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987053, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 20:24:13.116905 32686 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 20:24:13.137673 32686 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 20:24:14.113320 32686 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 20:24:14.113498 32686 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ranger-kms
I20260620 20:24:14.113655 32686 mini_ranger_kms.cc:227] Using postgres at 127.31.235.148:48935
2026-06-20 20:24:14,535 [I] DB FLAVOR :POSTGRES
2026-06-20 20:24:14,535 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 20:24:14,536 [I] Checking connection
2026-06-20 20:24:14,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 20:24:14,774 [I] connection success
2026-06-20 20:24:14,774 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 20:24:14,774 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 20:24:14,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 20:24:15,026 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 20:24:15,026 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 20:24:15,026 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 20:24:15,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:48935/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 20:24:15,313 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 20:24:15.768250 32686 mini_ranger_kms.cc:310] Created kms service
I20260620 20:24:15.979485 32686 mini_ranger_kms.cc:326] Created kudu user
I20260620 20:24:16.026082 32686 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 20:24:16 dist-test-slave-zzwf krb5kdc[11067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987053, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
I20260620 20:24:16.394245 32686 mini_ranger_kms.cc:384] Added ranger policy
I20260620 20:24:16.394750 32686 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ranger-kms:/tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 20:24:16.394814 32686 mini_ranger_kms.cc:240] Using host: 127.31.235.148
I20260620 20:24:16.411710 32686 mini_ranger_kms.cc:276] Ranger KMS PID: 11460
I20260620 20:24:16.411906 32686 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.235.148:42733
Jun 20, 2026 8:24:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:24:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:24:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 8:24:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 8:24:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 8:24:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 20:24:18.742851 32686 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.626s user 0.137s sys 0.298s
I20260620 20:24:18.743104 32686 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 20:24:18.743319 32686 mini_ranger_kms.cc:401] 127.31.235.148:42733/kms/v1/keys
Jun 20 20:24:18 dist-test-slave-zzwf krb5kdc[11067](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781987058, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.235.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 20:24:19 dist-test-slave-zzwf krb5kdc[11067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987058, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.235.148@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
Jun 20 20:24:19 dist-test-slave-zzwf krb5kdc[11067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987058, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.235.148@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
Jun 20 20:24:21 dist-test-slave-zzwf krb5kdc[11067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:24:21.881747 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:36671
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:36671
--ranger_config_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.235.148:42733/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:24:22.235277 11511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:24:22.235553 11511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:24:22.235689 11511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:24:22.242954 11511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:24:22.243070 11511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:24:22.243147 11511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:24:22.243245 11511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:24:22.243311 11511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:24:22.258505 11511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.235.148:42733/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:36671
--ranger_config_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:36671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:24:22.259877 11511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:24:22.261257 11511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:24:22.275100 11516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:24:22.275080 11517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:24:22.275044 11519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:24:22.275969 11511 server_base.cc:1034] running on GCE node
I20260620 20:24:22.276957 11511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:24:22.278826 11511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:24:22.280100 11511 hybrid_clock.cc:648] HybridClock initialized: now 1781987062280040 us; error 51 us; skew 500 ppm
I20260620 20:24:22.280496 11511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:24:22 dist-test-slave-zzwf krb5kdc[11067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987062, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:24:22.284368 11511 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:24:22.286007 11511 webserver.cc:460] Webserver started at http://127.31.235.190:36293/ using document root <none> and password file <none>
I20260620 20:24:22.286670 11511 fs_manager.cc:359] Metadata directory not provided
I20260620 20:24:22.286809 11511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:24:22.287112 11511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:24:22 dist-test-slave-zzwf krb5kdc[11067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987062, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
I20260620 20:24:22.345021 11511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "cd3c2b104b6f40088da0eb2c58dd3006"
format_stamp: "Formatted at 2026-06-20 20:24:22 on dist-test-slave-zzwf"
server_key: "f39260ec3f558a5aff2e6d0d31d9e766"
server_key_iv: "4074c44f2cae20f1b4a17d1fd90eab5c"
server_key_version: "kuduclusterkey@0"
I20260620 20:24:22.346099 11511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "cd3c2b104b6f40088da0eb2c58dd3006"
format_stamp: "Formatted at 2026-06-20 20:24:22 on dist-test-slave-zzwf"
server_key: "f39260ec3f558a5aff2e6d0d31d9e766"
server_key_iv: "4074c44f2cae20f1b4a17d1fd90eab5c"
server_key_version: "kuduclusterkey@0"
I20260620 20:24:22.352947 11511 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 20:24:22.372478 11526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:24:22.374245 11511 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 20:24:22.374449 11511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "cd3c2b104b6f40088da0eb2c58dd3006"
format_stamp: "Formatted at 2026-06-20 20:24:22 on dist-test-slave-zzwf"
server_key: "f39260ec3f558a5aff2e6d0d31d9e766"
server_key_iv: "4074c44f2cae20f1b4a17d1fd90eab5c"
server_key_version: "kuduclusterkey@0"
I20260620 20:24:22.374698 11511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:24:22.394151 11511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:24:22.399166 11511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:24:22.399510 11511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:24:22.419562 11511 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:36671
I20260620 20:24:22.419564 11577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:36671 every 8 connection(s)
I20260620 20:24:22.421308 11511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:24:22.425130 11578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:24:22.430820 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 11511
I20260620 20:24:22.431100 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:24:22.431897 32686 external_mini_cluster.cc:1428] Setting key d9b84ac6157fa070d50447271bf3cd4c
I20260620 20:24:22.438582 11578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006: Bootstrap starting.
I20260620 20:24:22.443815 11578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006: Neither blocks nor log segments found. Creating new log.
I20260620 20:24:22.445611 11578 log.cc:824] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006: Log is configured to *not* fsync() on all Append() calls
I20260620 20:24:22.451731 11578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006: No bootstrap required, opened a new log
I20260620 20:24:22.461086 11578 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3c2b104b6f40088da0eb2c58dd3006" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36671 } }
Jun 20 20:24:22 dist-test-slave-zzwf krb5kdc[11067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:24:22.461597 11578 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:24:22.461741 11578 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3c2b104b6f40088da0eb2c58dd3006, State: Initialized, Role: FOLLOWER
I20260620 20:24:22.462416 11578 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3c2b104b6f40088da0eb2c58dd3006" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36671 } }
I20260620 20:24:22.462718 11578 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:24:22.462877 11578 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:24:22.463093 11578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:24:22.466629 11578 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3c2b104b6f40088da0eb2c58dd3006" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36671 } }
I20260620 20:24:22.467286 11578 leader_election.cc:302] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd3c2b104b6f40088da0eb2c58dd3006; no voters:
I20260620 20:24:22.467958 11578 leader_election.cc:288] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:24:22.468096 11583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:24:22.469573 11583 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006 [term 1 LEADER]: Becoming Leader. State: Replica: cd3c2b104b6f40088da0eb2c58dd3006, State: Running, Role: LEADER
I20260620 20:24:22.470300 11578 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:24:22.470199 11583 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3c2b104b6f40088da0eb2c58dd3006" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36671 } }
I20260620 20:24:22.472620 11581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:22.434067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52360 (local address 127.31.235.190:36671)
0620 20:24:22.448862 (+ 14795us) server_negotiation.cc:206] Beginning negotiation
0620 20:24:22.448883 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 20:24:22.448941 (+ 58us) server_negotiation.cc:407] Connection header received
0620 20:24:22.451366 (+ 2425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:24:22.451394 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:24:22.451936 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:24:22.452392 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:24:22.453582 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:22.455748 (+ 2166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:22.457083 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:22.458218 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:22.458412 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:22.462694 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:24:22.462709 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:24:22.462733 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:24:22.462786 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:24:22.466727 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:24:22.467817 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:24:22.467830 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:24:22.467843 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:24:22.468056 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:24:22.468560 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:24:22.468566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:24:22.468572 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:24:22.469253 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:24:22.469604 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0620 20:24:22.470612 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0620 20:24:22.471060 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14438,"thread_start_us":14611,"threads_started":1}
I20260620 20:24:22.477332 11584 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd3c2b104b6f40088da0eb2c58dd3006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3c2b104b6f40088da0eb2c58dd3006" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36671 } } }
I20260620 20:24:22.477311 11585 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd3c2b104b6f40088da0eb2c58dd3006. Latest consensus state: current_term: 1 leader_uuid: "cd3c2b104b6f40088da0eb2c58dd3006" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3c2b104b6f40088da0eb2c58dd3006" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 36671 } } }
I20260620 20:24:22.477957 11585 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006 [sys.catalog]: This master's current role is: LEADER
I20260620 20:24:22.477989 11584 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006 [sys.catalog]: This master's current role is: LEADER
I20260620 20:24:22.480851 11596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:24:22.481563 11578 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 20:24:22.497781 11596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:24:22.511879 11596 catalog_manager.cc:1347] Generated new cluster ID: 24decd9e7bdb46eba84da98f96b77e1d
I20260620 20:24:22.512018 11596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:24:22.542284 11596 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:24:22.544608 11596 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:24:22.558562 11596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006: Generated new TSK 0
I20260620 20:24:22.559444 11596 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
I20260620 20:24:23.298501 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.235.148:42733/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36671
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:24:23 dist-test-slave-zzwf krb5kdc[11067](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781987063, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 20:24:23.738394 11623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:24:23.738915 11623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:24:23.739135 11623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:24:23.751274 11623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:24:23.751502 11623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:24:23.751734 11623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:24:23.767927 11623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.235.148:42733/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:24:23.769891 11623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:24:23.771715 11623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:24:23.790325 11633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:24:23.792189 11634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:24:23.798516 11623 server_base.cc:1034] running on GCE node
W20260620 20:24:23.798830 11636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:24:23.799633 11623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:24:23.804648 11623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:24:23.805968 11623 hybrid_clock.cc:648] HybridClock initialized: now 1781987063805863 us; error 95 us; skew 500 ppm
I20260620 20:24:23.806396 11623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:24:23 dist-test-slave-zzwf krb5kdc[11067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987063, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:24:23.811296 11623 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:24:23.813503 11623 webserver.cc:460] Webserver started at http://127.31.235.129:34267/ using document root <none> and password file <none>
I20260620 20:24:23.814435 11623 fs_manager.cc:359] Metadata directory not provided
I20260620 20:24:23.814574 11623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:24:23.814956 11623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:24:23 dist-test-slave-zzwf krb5kdc[11067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987063, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
I20260620 20:24:23.834211 11623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "4d3978a5769f47c390980f92ec17ee7e"
format_stamp: "Formatted at 2026-06-20 20:24:23 on dist-test-slave-zzwf"
server_key: "36245d471c20c9be0740ea73d3801052"
server_key_iv: "f031441b34d6feeca18adcd1aed30dd3"
server_key_version: "kuduclusterkey@0"
I20260620 20:24:23.835315 11623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "4d3978a5769f47c390980f92ec17ee7e"
format_stamp: "Formatted at 2026-06-20 20:24:23 on dist-test-slave-zzwf"
server_key: "36245d471c20c9be0740ea73d3801052"
server_key_iv: "f031441b34d6feeca18adcd1aed30dd3"
server_key_version: "kuduclusterkey@0"
I20260620 20:24:23.863601 11623 fs_manager.cc:691] Time spent creating directory manager: real 0.027s user 0.010s sys 0.015s
I20260620 20:24:23.886221 11643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:24:23.888008 11623 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.004s sys 0.004s
I20260620 20:24:23.888212 11623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "4d3978a5769f47c390980f92ec17ee7e"
format_stamp: "Formatted at 2026-06-20 20:24:23 on dist-test-slave-zzwf"
server_key: "36245d471c20c9be0740ea73d3801052"
server_key_iv: "f031441b34d6feeca18adcd1aed30dd3"
server_key_version: "kuduclusterkey@0"
I20260620 20:24:23.888466 11623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:24:23.908349 11623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:24:23.915735 11623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:24:23.916136 11623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:24:23.919315 11623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:24:23.921383 11623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:24:23.921542 11623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 20:24:23.921742 11623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:24:23.921859 11623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:24:24.028801 11623 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:36669
I20260620 20:24:24.028769 11755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:36669 every 8 connection(s)
I20260620 20:24:24.030658 11623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:24:24.039212 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 11623
I20260620 20:24:24.039449 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:24:24.040511 32686 external_mini_cluster.cc:1428] Setting key 1c0e776d360ae3942d6ac059f9aa3a78
Jun 20 20:24:24 dist-test-slave-zzwf krb5kdc[11067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987063, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.235.130@KRBTEST.COM; defaulting to no policy
I20260620 20:24:24.077965 11758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:24.039717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:47893 (local address 127.31.235.190:36671)
0620 20:24:24.053868 (+ 14151us) server_negotiation.cc:206] Beginning negotiation
0620 20:24:24.053874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:24:24.053896 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:24:24.053962 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:24:24.053966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:24:24.054066 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:24:24.054189 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:24:24.057665 (+ 3476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:24.058954 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:24.061679 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:24.062419 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:24.062548 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:24.069744 (+ 7196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:24:24.069751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:24:24.069756 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:24:24.069812 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:24:24.073718 (+ 3906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:24:24.074818 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:24:24.074824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:24:24.074828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:24:24.074973 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:24:24.075571 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:24:24.075576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:24:24.075579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:24:24.075948 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:24:24.076087 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 20:24:24.077165 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0620 20:24:24.077432 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14015,"thread_start_us":16627,"threads_started":1}
I20260620 20:24:24.078896 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:24.040363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:36671 (local address 127.31.235.129:47893)
0620 20:24:24.045276 (+ 4913us) negotiation.cc:107] Waiting for socket to connect
0620 20:24:24.045349 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 20:24:24.047984 (+ 2635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:24:24.054682 (+ 6698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:24:24.054695 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:24:24.055376 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:24:24.056178 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:24:24.056204 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:24:24.059172 (+ 2968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:24:24.059178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:24:24.061419 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:24:24.061436 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:24:24.062878 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:24:24.062886 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:24:24.063205 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:24.064529 (+ 1324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:24:24.064561 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:24:24.068182 (+ 3621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:24:24.073901 (+ 5719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:24:24.073911 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:24:24.073932 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:24:24.074662 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:24:24.075114 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:24:24.075119 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:24:24.075122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:24:24.075383 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:24:24.076164 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:24:24.076172 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:24:24.076687 (+ 515us) client_negotiation.cc:769] Sending connection context
0620 20:24:24.077020 (+ 333us) client_negotiation.cc:240] Negotiation successful
0620 20:24:24.077619 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4450,"thread_start_us":12162,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.130@KRBTEST.COM" created.
I20260620 20:24:24.080696 11756 heartbeater.cc:347] Connected to a master server at 127.31.235.190:36671
I20260620 20:24:24.081020 11756 heartbeater.cc:464] Registering TS with master...
I20260620 20:24:24.081964 11756 heartbeater.cc:511] Master 127.31.235.190:36671 requested a full tablet report, sending...
I20260620 20:24:24.084954 11543 ts_manager.cc:194] Registered new tserver with Master: 4d3978a5769f47c390980f92ec17ee7e (127.31.235.129:36669)
I20260620 20:24:24.087015 11543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:47893
Jun 20 20:24:24 dist-test-slave-zzwf krb5kdc[11067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987063, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
I20260620 20:24:24.123447 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.235.148:42733/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36671
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:24:24 dist-test-slave-zzwf krb5kdc[11067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987063, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
W20260620 20:24:24.658082 11762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:24:24.658595 11762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:24:24.658793 11762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:24:24.670619 11762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:24:24.670850 11762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:24:24.671072 11762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:24:24.686421 11762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.235.148:42733/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:24:24.689137 11762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:24:24.690963 11762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:24:24.735028 11769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:24:24.739853 11770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:24:24.744613 11773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:24:24.745162 11762 server_base.cc:1034] running on GCE node
I20260620 20:24:24.746194 11762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:24:24.752787 11762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:24:24.754158 11762 hybrid_clock.cc:648] HybridClock initialized: now 1781987064754051 us; error 108 us; skew 500 ppm
I20260620 20:24:24.754647 11762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:24:24 dist-test-slave-zzwf krb5kdc[11067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987064, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:24:24.759264 11762 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:24:24.764066 11762 webserver.cc:460] Webserver started at http://127.31.235.130:41443/ using document root <none> and password file <none>
I20260620 20:24:24.764998 11762 fs_manager.cc:359] Metadata directory not provided
I20260620 20:24:24.765131 11762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:24:24.765467 11762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:24:24 dist-test-slave-zzwf krb5kdc[11067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987064, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
I20260620 20:24:24.805224 11762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "b7232edb1924496b97a36e2e1b993a0d"
format_stamp: "Formatted at 2026-06-20 20:24:24 on dist-test-slave-zzwf"
server_key: "7710cb37a7f98a460f04bfe4aea95768"
server_key_iv: "1dce1b913fea4accab7b4740e488aee9"
server_key_version: "kuduclusterkey@0"
I20260620 20:24:24.808630 11762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "b7232edb1924496b97a36e2e1b993a0d"
format_stamp: "Formatted at 2026-06-20 20:24:24 on dist-test-slave-zzwf"
server_key: "7710cb37a7f98a460f04bfe4aea95768"
server_key_iv: "1dce1b913fea4accab7b4740e488aee9"
server_key_version: "kuduclusterkey@0"
I20260620 20:24:24.823316 11762 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.003s sys 0.008s
I20260620 20:24:24.854367 11779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:24:24.856308 11762 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.004s
I20260620 20:24:24.856586 11762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "b7232edb1924496b97a36e2e1b993a0d"
format_stamp: "Formatted at 2026-06-20 20:24:24 on dist-test-slave-zzwf"
server_key: "7710cb37a7f98a460f04bfe4aea95768"
server_key_iv: "1dce1b913fea4accab7b4740e488aee9"
server_key_version: "kuduclusterkey@0"
I20260620 20:24:24.856943 11762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:24:24.903762 11762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:24:24.907981 11762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:24:24.908347 11762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:24:24.909863 11762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:24:24.911350 11762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:24:24.911453 11762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:24:24.911592 11762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:24:24.911664 11762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:24:24.953600 11762 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:35419
I20260620 20:24:24.953615 11898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:35419 every 8 connection(s)
I20260620 20:24:24.954900 11762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:24:24.956326 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 11762
I20260620 20:24:24.956532 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:24:24.957291 32686 external_mini_cluster.cc:1428] Setting key 5d3ae11d8dd3a06c252e95ce84837d42
WARNING: no policy specified for kudu/127.31.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.131@KRBTEST.COM" created.
Jun 20 20:24:24 dist-test-slave-zzwf krb5kdc[11067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987064, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:24:24.993865 11901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:24.958212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:35607 (local address 127.31.235.190:36671)
0620 20:24:24.969197 (+ 10985us) server_negotiation.cc:206] Beginning negotiation
0620 20:24:24.969204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:24:24.970347 (+ 1143us) server_negotiation.cc:407] Connection header received
0620 20:24:24.972971 (+ 2624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:24:24.972980 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:24:24.973086 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:24:24.973215 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:24:24.975171 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:24.976373 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:24.978742 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:24.979549 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:24.979702 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:24.987235 (+ 7533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:24:24.987241 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:24:24.987245 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:24:24.987280 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:24:24.990398 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:24:24.991159 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:24:24.991176 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:24:24.991178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:24:24.991359 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:24:24.991832 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:24:24.991838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:24:24.991842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:24:24.992140 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:24:24.992318 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 20:24:24.993180 (+ 862us) server_negotiation.cc:299] Negotiation successful
0620 20:24:24.993430 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10855,"thread_start_us":11309,"threads_started":1}
I20260620 20:24:24.995311 11902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:24.959122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:36671 (local address 127.31.235.130:35607)
0620 20:24:24.970160 (+ 11038us) negotiation.cc:107] Waiting for socket to connect
0620 20:24:24.970222 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 20:24:24.972651 (+ 2429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:24:24.973474 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:24:24.973511 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:24:24.974175 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:24:24.974983 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:24:24.975005 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:24:24.976546 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:24:24.976552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:24:24.978584 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:24:24.978598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:24:24.979819 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:24:24.979826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:24:24.980133 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:24.980942 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:24:24.980972 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:24:24.986959 (+ 5987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:24:24.990557 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:24:24.990566 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:24:24.990591 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:24:24.991020 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:24:24.991479 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:24:24.991485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:24:24.991489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:24:24.991701 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:24:24.992283 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:24:24.992291 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:24:24.992771 (+ 480us) client_negotiation.cc:769] Sending connection context
0620 20:24:24.993047 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 20:24:24.993641 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10738,"thread_start_us":10465,"threads_started":1}
I20260620 20:24:24.996937 11899 heartbeater.cc:347] Connected to a master server at 127.31.235.190:36671
I20260620 20:24:24.997326 11899 heartbeater.cc:464] Registering TS with master...
I20260620 20:24:24.998261 11899 heartbeater.cc:511] Master 127.31.235.190:36671 requested a full tablet report, sending...
I20260620 20:24:25.000411 11543 ts_manager.cc:194] Registered new tserver with Master: b7232edb1924496b97a36e2e1b993a0d (127.31.235.130:35419)
I20260620 20:24:25.001823 11543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:35607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
I20260620 20:24:25.021597 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.235.148:42733/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36671
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:40197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:24:25.090773 11756 heartbeater.cc:503] Master 127.31.235.190:36671 was elected leader, sending a full tablet report...
W20260620 20:24:25.362373 11905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:24:25.362730 11905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:24:25.362926 11905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:24:25.370524 11905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:24:25.370668 11905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:24:25.370858 11905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:24:25.381484 11905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:40197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.235.148:42733/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:36671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:24:25.382829 11905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:24:25.384078 11905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:24:25.397616 11910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:24:25.397660 11913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:24:25.397706 11911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:24:25.403076 11905 server_base.cc:1034] running on GCE node
I20260620 20:24:25.403713 11905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:24:25.405105 11905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:24:25.406314 11905 hybrid_clock.cc:648] HybridClock initialized: now 1781987065406256 us; error 51 us; skew 500 ppm
I20260620 20:24:25.406586 11905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:24:25 dist-test-slave-zzwf krb5kdc[11067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987065, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:24:25.409477 11905 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:24:25.410991 11905 webserver.cc:460] Webserver started at http://127.31.235.131:39159/ using document root <none> and password file <none>
I20260620 20:24:25.411623 11905 fs_manager.cc:359] Metadata directory not provided
I20260620 20:24:25.411717 11905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:24:25.411957 11905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:24:25 dist-test-slave-zzwf krb5kdc[11067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987065, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
I20260620 20:24:25.427078 11905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "6308757430844b739f525db7b569f9bf"
format_stamp: "Formatted at 2026-06-20 20:24:25 on dist-test-slave-zzwf"
server_key: "23c0a512923f036c2db165fe021658ba"
server_key_iv: "6b16321561631a8e01b66399dd2de308"
server_key_version: "kuduclusterkey@0"
I20260620 20:24:25.428057 11905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "6308757430844b739f525db7b569f9bf"
format_stamp: "Formatted at 2026-06-20 20:24:25 on dist-test-slave-zzwf"
server_key: "23c0a512923f036c2db165fe021658ba"
server_key_iv: "6b16321561631a8e01b66399dd2de308"
server_key_version: "kuduclusterkey@0"
I20260620 20:24:25.434619 11905 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260620 20:24:25.452658 11920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:24:25.454385 11905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 20:24:25.454578 11905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "6308757430844b739f525db7b569f9bf"
format_stamp: "Formatted at 2026-06-20 20:24:25 on dist-test-slave-zzwf"
server_key: "23c0a512923f036c2db165fe021658ba"
server_key_iv: "6b16321561631a8e01b66399dd2de308"
server_key_version: "kuduclusterkey@0"
I20260620 20:24:25.454828 11905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:24:25.475064 11905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:24:25.479864 11905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:24:25.480170 11905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:24:25.481721 11905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:24:25.483317 11905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:24:25.483417 11905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:24:25.483521 11905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:24:25.483584 11905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:24:25.519212 11905 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:39813
I20260620 20:24:25.519244 12032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:39813 every 8 connection(s)
I20260620 20:24:25.520529 11905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:24:25.526811 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 11905
I20260620 20:24:25.526999 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:24:25.527403 32686 external_mini_cluster.cc:1428] Setting key 09ea8f38b8152946079b4fd4283c7290
Jun 20 20:24:25 dist-test-slave-zzwf krb5kdc[11067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987065, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:24:25.540490 12035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:25.523078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:41349 (local address 127.31.235.190:36671)
0620 20:24:25.523559 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 20:24:25.523565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:24:25.524419 (+ 854us) server_negotiation.cc:407] Connection header received
0620 20:24:25.526154 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:24:25.526160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:24:25.526244 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:24:25.526339 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:24:25.528559 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.529622 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.531587 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.531989 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.532081 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:25.535374 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:24:25.535379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:24:25.535382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:24:25.535415 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:24:25.537880 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:24:25.538561 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:24:25.538565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:24:25.538567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:24:25.538683 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:24:25.539074 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:24:25.539077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:24:25.539078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:24:25.539319 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:24:25.539432 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 20:24:25.540070 (+ 638us) server_negotiation.cc:299] Negotiation successful
0620 20:24:25.540218 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":208,"threads_started":1}
I20260620 20:24:25.541893 12036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:25.523542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:36671 (local address 127.31.235.131:41349)
0620 20:24:25.524250 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0620 20:24:25.524286 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:24:25.525916 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:24:25.526629 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:24:25.526637 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:24:25.527236 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:24:25.528342 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:24:25.528368 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.529774 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.529780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:24:25.531446 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:24:25.531466 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.532173 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.532178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:24:25.532384 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:25.533013 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:24:25.533036 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:24:25.535158 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:24:25.538002 (+ 2844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:24:25.538009 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:24:25.538022 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:24:25.538462 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:24:25.538776 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:24:25.538779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:24:25.538782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:24:25.538977 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:24:25.539439 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:24:25.539445 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:24:25.539742 (+ 297us) client_negotiation.cc:769] Sending connection context
0620 20:24:25.539999 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 20:24:25.540385 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":285,"threads_started":1}
I20260620 20:24:25.543367 12033 heartbeater.cc:347] Connected to a master server at 127.31.235.190:36671
I20260620 20:24:25.543627 12033 heartbeater.cc:464] Registering TS with master...
I20260620 20:24:25.544301 12033 heartbeater.cc:511] Master 127.31.235.190:36671 requested a full tablet report, sending...
I20260620 20:24:25.545665 11542 ts_manager.cc:194] Registered new tserver with Master: 6308757430844b739f525db7b569f9bf (127.31.235.131:39813)
I20260620 20:24:25.546537 11542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:41349
I20260620 20:24:25.555450 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:24:25.572805 12035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:25.559794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52364 (local address 127.31.235.190:36671)
0620 20:24:25.560015 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 20:24:25.560021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:24:25.560817 (+ 796us) server_negotiation.cc:407] Connection header received
0620 20:24:25.561042 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:24:25.561063 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:24:25.561153 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:24:25.561236 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:24:25.562666 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.563369 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.564621 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.565104 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.565192 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:25.567664 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:24:25.567669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:24:25.567673 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:24:25.567706 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:24:25.570241 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:24:25.570960 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:24:25.570964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:24:25.570966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:24:25.571092 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:24:25.571432 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:24:25.571436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:24:25.571438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:24:25.571800 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:24:25.572001 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 20:24:25.572291 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 20:24:25.572510 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:24:25.576949 11542 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:24:25.584426 11542 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:24:25.607962 12046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:25.597153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:35419 (local address 127.0.0.1:53476)
0620 20:24:25.597982 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0620 20:24:25.598025 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:24:25.598194 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:24:25.600430 (+ 2236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:24:25.600436 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:24:25.600468 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:24:25.600863 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:24:25.600875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.602912 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.602918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:24:25.605049 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:24:25.605058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.606830 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.606837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:24:25.607012 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:25.607049 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 20:24:25.607113 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 20:24:25.607237 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":395,"threads_started":1}
I20260620 20:24:25.608212 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:25.598021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53476 (local address 127.31.235.130:35419)
0620 20:24:25.599417 (+ 1396us) server_negotiation.cc:206] Beginning negotiation
0620 20:24:25.599426 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:24:25.599453 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:24:25.599577 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:24:25.599584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:24:25.600209 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:24:25.601054 (+ 845us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:24:25.601267 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.602728 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.605190 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.606596 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.607431 (+ 835us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:25.607627 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 20:24:25.607700 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 20:24:25.607866 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1245,"thread_start_us":1134,"threads_started":1}
I20260620 20:24:25.607985 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:25.597560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:39813 (local address 127.0.0.1:38648)
0620 20:24:25.598220 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0620 20:24:25.598240 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:24:25.598353 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:24:25.599609 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:24:25.599618 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:24:25.599656 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:24:25.600059 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:24:25.600071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.602184 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.602192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:24:25.604808 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:24:25.604821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.606652 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.606658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:24:25.606852 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:25.606894 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 20:24:25.606979 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 20:24:25.607103 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":4,"thread_start_us":777,"threads_started":1}
I20260620 20:24:25.610129 12050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:25.599306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:36669 (local address 127.0.0.1:49682)
0620 20:24:25.602493 (+ 3187us) negotiation.cc:107] Waiting for socket to connect
0620 20:24:25.602509 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:24:25.602630 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:24:25.603352 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:24:25.603357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:24:25.603395 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:24:25.603905 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:24:25.603913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.605526 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.605531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:24:25.607766 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:24:25.607779 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.609338 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.609344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:24:25.609548 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:25.609579 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 20:24:25.609700 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 20:24:25.609819 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3043,"thread_start_us":2888,"threads_started":1}
I20260620 20:24:25.610268 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:25.599958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49682 (local address 127.31.235.129:36669)
0620 20:24:25.602011 (+ 2053us) server_negotiation.cc:206] Beginning negotiation
0620 20:24:25.602022 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:24:25.602761 (+ 739us) server_negotiation.cc:407] Connection header received
0620 20:24:25.602879 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:24:25.602887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:24:25.603160 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:24:25.603372 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:24:25.604081 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.605352 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.607967 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.609118 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.609330 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:25.609485 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 20:24:25.609749 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 20:24:25.609918 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1851,"thread_start_us":1709,"threads_started":1}
I20260620 20:24:25.611577 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:25.598368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38648 (local address 127.31.235.131:39813)
0620 20:24:25.598942 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0620 20:24:25.598950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:24:25.598984 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:24:25.599089 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:24:25.599100 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:24:25.599358 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:24:25.599593 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:24:25.600375 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.602021 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.604971 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.606427 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.608675 (+ 2248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:25.608834 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 20:24:25.608901 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 20:24:25.609061 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":272,"threads_started":1}
I20260620 20:24:25.612191 11834 tablet_service.cc:1467] Processing CreateTablet for tablet fa0d5100cdbd4de1a88fca7b866557ab (DEFAULT_TABLE table=test-table [id=c29eb9841a0b4b778fc616f33c35cfad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:24:25.613812 11834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa0d5100cdbd4de1a88fca7b866557ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:24:25.614168 11691 tablet_service.cc:1467] Processing CreateTablet for tablet fa0d5100cdbd4de1a88fca7b866557ab (DEFAULT_TABLE table=test-table [id=c29eb9841a0b4b778fc616f33c35cfad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:24:25.614418 11968 tablet_service.cc:1467] Processing CreateTablet for tablet fa0d5100cdbd4de1a88fca7b866557ab (DEFAULT_TABLE table=test-table [id=c29eb9841a0b4b778fc616f33c35cfad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:24:25.615621 11691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa0d5100cdbd4de1a88fca7b866557ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:24:25.615720 11968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa0d5100cdbd4de1a88fca7b866557ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:24:25.630154 12053 tablet_bootstrap.cc:492] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d: Bootstrap starting.
I20260620 20:24:25.632560 12052 tablet_bootstrap.cc:492] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf: Bootstrap starting.
I20260620 20:24:25.634218 12053 tablet_bootstrap.cc:654] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d: Neither blocks nor log segments found. Creating new log.
I20260620 20:24:25.634198 12054 tablet_bootstrap.cc:492] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e: Bootstrap starting.
I20260620 20:24:25.635584 12053 log.cc:824] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d: Log is configured to *not* fsync() on all Append() calls
I20260620 20:24:25.636078 12052 tablet_bootstrap.cc:654] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf: Neither blocks nor log segments found. Creating new log.
I20260620 20:24:25.637315 12052 log.cc:824] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf: Log is configured to *not* fsync() on all Append() calls
I20260620 20:24:25.638223 12054 tablet_bootstrap.cc:654] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e: Neither blocks nor log segments found. Creating new log.
I20260620 20:24:25.639264 12053 tablet_bootstrap.cc:492] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d: No bootstrap required, opened a new log
I20260620 20:24:25.639509 12054 log.cc:824] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:24:25.639534 12053 ts_tablet_manager.cc:1397] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260620 20:24:25.640754 12052 tablet_bootstrap.cc:492] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf: No bootstrap required, opened a new log
I20260620 20:24:25.641021 12052 ts_tablet_manager.cc:1397] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 20:24:25.642068 12054 tablet_bootstrap.cc:492] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e: No bootstrap required, opened a new log
I20260620 20:24:25.642315 12054 ts_tablet_manager.cc:1397] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 20:24:25.649817 12054 raft_consensus.cc:348] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7232edb1924496b97a36e2e1b993a0d" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35419 } } peers { permanent_uuid: "6308757430844b739f525db7b569f9bf" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 39813 } } peers { permanent_uuid: "4d3978a5769f47c390980f92ec17ee7e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36669 } }
I20260620 20:24:25.650203 12054 raft_consensus.cc:374] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:24:25.650364 12054 raft_consensus.cc:729] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3978a5769f47c390980f92ec17ee7e, State: Initialized, Role: FOLLOWER
I20260620 20:24:25.650458 12053 raft_consensus.cc:348] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7232edb1924496b97a36e2e1b993a0d" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35419 } } peers { permanent_uuid: "6308757430844b739f525db7b569f9bf" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 39813 } } peers { permanent_uuid: "4d3978a5769f47c390980f92ec17ee7e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36669 } }
I20260620 20:24:25.650856 12053 raft_consensus.cc:374] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:24:25.650964 12053 raft_consensus.cc:729] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7232edb1924496b97a36e2e1b993a0d, State: Initialized, Role: FOLLOWER
I20260620 20:24:25.651057 12054 consensus_queue.cc:260] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7232edb1924496b97a36e2e1b993a0d" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35419 } } peers { permanent_uuid: "6308757430844b739f525db7b569f9bf" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 39813 } } peers { permanent_uuid: "4d3978a5769f47c390980f92ec17ee7e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36669 } }
I20260620 20:24:25.651513 12053 consensus_queue.cc:260] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7232edb1924496b97a36e2e1b993a0d" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35419 } } peers { permanent_uuid: "6308757430844b739f525db7b569f9bf" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 39813 } } peers { permanent_uuid: "4d3978a5769f47c390980f92ec17ee7e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36669 } }
I20260620 20:24:25.651628 12052 raft_consensus.cc:348] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7232edb1924496b97a36e2e1b993a0d" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35419 } } peers { permanent_uuid: "6308757430844b739f525db7b569f9bf" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 39813 } } peers { permanent_uuid: "4d3978a5769f47c390980f92ec17ee7e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36669 } }
I20260620 20:24:25.652037 12052 raft_consensus.cc:374] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:24:25.652211 12052 raft_consensus.cc:729] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6308757430844b739f525db7b569f9bf, State: Initialized, Role: FOLLOWER
I20260620 20:24:25.652951 12054 ts_tablet_manager.cc:1428] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260620 20:24:25.653065 12052 consensus_queue.cc:260] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7232edb1924496b97a36e2e1b993a0d" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35419 } } peers { permanent_uuid: "6308757430844b739f525db7b569f9bf" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 39813 } } peers { permanent_uuid: "4d3978a5769f47c390980f92ec17ee7e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36669 } }
I20260620 20:24:25.656267 12033 heartbeater.cc:503] Master 127.31.235.190:36671 was elected leader, sending a full tablet report...
I20260620 20:24:25.657341 12052 ts_tablet_manager.cc:1428] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260620 20:24:25.658401 11899 heartbeater.cc:503] Master 127.31.235.190:36671 was elected leader, sending a full tablet report...
I20260620 20:24:25.659483 12053 ts_tablet_manager.cc:1428] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d: Time spent starting tablet: real 0.020s user 0.021s sys 0.000s
W20260620 20:24:25.713759 11900 tablet.cc:2369] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:24:25.772200 12034 tablet.cc:2369] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:24:25.802788 11757 tablet.cc:2369] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:24:25.804293 12059 raft_consensus.cc:482] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:24:25.804603 12059 raft_consensus.cc:504] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7232edb1924496b97a36e2e1b993a0d" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35419 } } peers { permanent_uuid: "6308757430844b739f525db7b569f9bf" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 39813 } } peers { permanent_uuid: "4d3978a5769f47c390980f92ec17ee7e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36669 } }
I20260620 20:24:25.806347 12059 leader_election.cc:288] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6308757430844b739f525db7b569f9bf (127.31.235.131:39813), 4d3978a5769f47c390980f92ec17ee7e (127.31.235.129:36669)
I20260620 20:24:25.813915 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:25.807288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:37823 (local address 127.31.235.129:36669)
0620 20:24:25.807515 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 20:24:25.807523 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:24:25.808585 (+ 1062us) server_negotiation.cc:407] Connection header received
0620 20:24:25.808809 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:24:25.808815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:24:25.808898 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:24:25.808998 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:24:25.809582 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.810341 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.812177 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.812942 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.813093 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:25.813187 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 20:24:25.813539 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 20:24:25.813664 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:24:25.814030 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:25.807168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:57479 (local address 127.31.235.131:39813)
0620 20:24:25.807406 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 20:24:25.807415 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:24:25.808292 (+ 877us) server_negotiation.cc:407] Connection header received
0620 20:24:25.808521 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:24:25.808529 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:24:25.808605 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:24:25.808694 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:24:25.809336 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.810162 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.812206 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.812976 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.813163 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:25.813229 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 20:24:25.813656 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 20:24:25.813790 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:24:25.814070 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:25.807738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:36669 (local address 127.31.235.130:37823)
0620 20:24:25.808464 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 20:24:25.808481 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:24:25.808607 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:24:25.809030 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:24:25.809033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:24:25.809053 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:24:25.809404 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:24:25.809418 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.810487 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.810491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:24:25.812028 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:24:25.812038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.813173 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.813178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:24:25.813354 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:25.813379 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 20:24:25.813483 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 20:24:25.813622 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"mutex_wait_us":23,"thread_start_us":373,"threads_started":1}
I20260620 20:24:25.814167 12061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:25.807198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:39813 (local address 127.31.235.130:57479)
0620 20:24:25.808135 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0620 20:24:25.808160 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 20:24:25.808359 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:24:25.808727 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:24:25.808731 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:24:25.808759 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:24:25.809164 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:24:25.809190 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.810287 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.810294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:24:25.812032 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:24:25.812043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.813228 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.813241 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:24:25.813452 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:25.813498 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 20:24:25.813621 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 20:24:25.813747 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":120,"thread_start_us":545,"threads_started":1}
I20260620 20:24:25.815148 12058 raft_consensus.cc:482] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:24:25.815270 11988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa0d5100cdbd4de1a88fca7b866557ab" candidate_uuid: "b7232edb1924496b97a36e2e1b993a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6308757430844b739f525db7b569f9bf" is_pre_election: true
I20260620 20:24:25.815789 11988 raft_consensus.cc:2454] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7232edb1924496b97a36e2e1b993a0d in term 0.
I20260620 20:24:25.815392 11711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa0d5100cdbd4de1a88fca7b866557ab" candidate_uuid: "b7232edb1924496b97a36e2e1b993a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3978a5769f47c390980f92ec17ee7e" is_pre_election: true
I20260620 20:24:25.815459 12058 raft_consensus.cc:504] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7232edb1924496b97a36e2e1b993a0d" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35419 } } peers { permanent_uuid: "6308757430844b739f525db7b569f9bf" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 39813 } } peers { permanent_uuid: "4d3978a5769f47c390980f92ec17ee7e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36669 } }
I20260620 20:24:25.816556 11781 leader_election.cc:302] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6308757430844b739f525db7b569f9bf, b7232edb1924496b97a36e2e1b993a0d; no voters:
I20260620 20:24:25.816612 11711 raft_consensus.cc:2454] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7232edb1924496b97a36e2e1b993a0d in term 0.
I20260620 20:24:25.816912 12059 raft_consensus.cc:2781] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:24:25.817059 12059 raft_consensus.cc:482] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:24:25.817178 12059 raft_consensus.cc:3037] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:24:25.817827 12058 leader_election.cc:288] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7232edb1924496b97a36e2e1b993a0d (127.31.235.130:35419), 6308757430844b739f525db7b569f9bf (127.31.235.131:39813)
I20260620 20:24:25.821122 12059 raft_consensus.cc:504] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7232edb1924496b97a36e2e1b993a0d" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35419 } } peers { permanent_uuid: "6308757430844b739f525db7b569f9bf" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 39813 } } peers { permanent_uuid: "4d3978a5769f47c390980f92ec17ee7e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36669 } }
I20260620 20:24:25.821995 12059 leader_election.cc:288] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [CANDIDATE]: Term 1 election: Requested vote from peers 6308757430844b739f525db7b569f9bf (127.31.235.131:39813), 4d3978a5769f47c390980f92ec17ee7e (127.31.235.129:36669)
I20260620 20:24:25.822415 11988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa0d5100cdbd4de1a88fca7b866557ab" candidate_uuid: "b7232edb1924496b97a36e2e1b993a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6308757430844b739f525db7b569f9bf"
I20260620 20:24:25.822566 11711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa0d5100cdbd4de1a88fca7b866557ab" candidate_uuid: "b7232edb1924496b97a36e2e1b993a0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3978a5769f47c390980f92ec17ee7e"
I20260620 20:24:25.822631 11988 raft_consensus.cc:3037] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:24:25.822757 11711 raft_consensus.cc:3037] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:24:25.824448 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:25.818432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:39229 (local address 127.31.235.130:35419)
0620 20:24:25.818608 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 20:24:25.818615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:24:25.818702 (+ 87us) server_negotiation.cc:407] Connection header received
0620 20:24:25.818940 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:24:25.818945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:24:25.819012 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:24:25.819102 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:24:25.819585 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.820284 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.822105 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.823260 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.823380 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:25.823446 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 20:24:25.824062 (+ 616us) server_negotiation.cc:299] Negotiation successful
0620 20:24:25.824200 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:24:25.825083 12063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:25.817992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:35419 (local address 127.31.235.129:39229)
0620 20:24:25.818607 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0620 20:24:25.818631 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:24:25.818799 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:24:25.819129 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:24:25.819134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:24:25.819160 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:24:25.819463 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:24:25.819471 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.820418 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.820426 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:24:25.821927 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:24:25.821938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.823656 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.823665 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:24:25.823884 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:25.823928 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 20:24:25.824597 (+ 669us) client_negotiation.cc:240] Negotiation successful
0620 20:24:25.824738 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":631,"threads_started":1}
I20260620 20:24:25.825884 11854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa0d5100cdbd4de1a88fca7b866557ab" candidate_uuid: "4d3978a5769f47c390980f92ec17ee7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7232edb1924496b97a36e2e1b993a0d" is_pre_election: true
I20260620 20:24:25.826294 11854 raft_consensus.cc:2379] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d3978a5769f47c390980f92ec17ee7e in current term 1: Already voted for candidate b7232edb1924496b97a36e2e1b993a0d in this term.
I20260620 20:24:25.826464 11988 raft_consensus.cc:2454] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7232edb1924496b97a36e2e1b993a0d in term 1.
I20260620 20:24:25.826925 11781 leader_election.cc:302] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6308757430844b739f525db7b569f9bf, b7232edb1924496b97a36e2e1b993a0d; no voters:
I20260620 20:24:25.827194 12059 raft_consensus.cc:2781] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:24:25.827628 12059 raft_consensus.cc:686] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [term 1 LEADER]: Becoming Leader. State: Replica: b7232edb1924496b97a36e2e1b993a0d, State: Running, Role: LEADER
I20260620 20:24:25.828176 12059 consensus_queue.cc:237] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7232edb1924496b97a36e2e1b993a0d" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35419 } } peers { permanent_uuid: "6308757430844b739f525db7b569f9bf" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 39813 } } peers { permanent_uuid: "4d3978a5769f47c390980f92ec17ee7e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36669 } }
I20260620 20:24:25.828459 11711 raft_consensus.cc:2454] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7232edb1924496b97a36e2e1b993a0d in term 1.
I20260620 20:24:25.833324 12064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:25.818236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:39813 (local address 127.31.235.129:50331)
0620 20:24:25.822567 (+ 4331us) negotiation.cc:107] Waiting for socket to connect
0620 20:24:25.822585 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:24:25.822775 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:24:25.823295 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:24:25.823303 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:24:25.823333 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:24:25.823782 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:24:25.823796 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.825734 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.825739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:24:25.831089 (+ 5350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:24:25.831111 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.832772 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.832776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:24:25.832883 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:25.832904 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:24:25.832974 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 20:24:25.833063 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4174,"thread_start_us":8428,"threads_started":1}
I20260620 20:24:25.833417 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:25.818639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:50331 (local address 127.31.235.131:39813)
0620 20:24:25.818791 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 20:24:25.818796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:24:25.822659 (+ 3863us) server_negotiation.cc:407] Connection header received
0620 20:24:25.822940 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:24:25.822946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:24:25.823020 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:24:25.823084 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:24:25.824439 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.825552 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.831309 (+ 5757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.832468 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.832565 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:25.832613 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 20:24:25.833038 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 20:24:25.833165 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:24:25.833034 11542 catalog_manager.cc:5495] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d reported cstate change: term changed from 0 to 1, leader changed from <none> to b7232edb1924496b97a36e2e1b993a0d (127.31.235.130). New cstate: current_term: 1 leader_uuid: "b7232edb1924496b97a36e2e1b993a0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7232edb1924496b97a36e2e1b993a0d" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6308757430844b739f525db7b569f9bf" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3978a5769f47c390980f92ec17ee7e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36669 } health_report { overall_health: UNKNOWN } } }
I20260620 20:24:25.834255 11988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa0d5100cdbd4de1a88fca7b866557ab" candidate_uuid: "4d3978a5769f47c390980f92ec17ee7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6308757430844b739f525db7b569f9bf" is_pre_election: true
I20260620 20:24:25.834632 11988 raft_consensus.cc:2379] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d3978a5769f47c390980f92ec17ee7e in current term 1: Already voted for candidate b7232edb1924496b97a36e2e1b993a0d in this term.
I20260620 20:24:25.835500 11644 leader_election.cc:302] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3978a5769f47c390980f92ec17ee7e; no voters: 6308757430844b739f525db7b569f9bf, b7232edb1924496b97a36e2e1b993a0d
I20260620 20:24:25.836270 12058 raft_consensus.cc:2726] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 20:24:25.904405 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:25.898888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53484 (local address 127.31.235.130:35419)
0620 20:24:25.899092 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 20:24:25.899098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:24:25.899116 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:24:25.899286 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:24:25.899291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:24:25.899378 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:24:25.899538 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:24:25.899943 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.900660 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.902214 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.902626 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.902770 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:25.903304 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:24:25.903797 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:24:25.903979 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 20:24:25.904029 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 20:24:25.904109 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:24:25.912189 11711 raft_consensus.cc:1261] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e [term 1 FOLLOWER]: Refusing update from remote peer b7232edb1924496b97a36e2e1b993a0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:24:25.912205 11988 raft_consensus.cc:1261] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf [term 1 FOLLOWER]: Refusing update from remote peer b7232edb1924496b97a36e2e1b993a0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:24:25.913044 12065 consensus_queue.cc:1035] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3978a5769f47c390980f92ec17ee7e" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 36669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:24:25.913371 12059 consensus_queue.cc:1035] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6308757430844b739f525db7b569f9bf" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:24:25.926589 12068 mvcc.cc:205] Tried to move back new op lower bound from 7299019021967101952 to 7299019021636710400. Current Snapshot: MvccSnapshot[applied={T|T < 7299019021967101952}]
I20260620 20:24:25.941776 11542 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:24:25.942044 11542 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:24:25.945438 11542 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd3c2b104b6f40088da0eb2c58dd3006: Sending DeleteTablet for 3 replicas of tablet fa0d5100cdbd4de1a88fca7b866557ab
I20260620 20:24:25.946576 11834 tablet_service.cc:1514] Processing DeleteTablet for tablet fa0d5100cdbd4de1a88fca7b866557ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:24:25 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:53476
I20260620 20:24:25.946595 11968 tablet_service.cc:1514] Processing DeleteTablet for tablet fa0d5100cdbd4de1a88fca7b866557ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:24:25 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:38648
I20260620 20:24:25.946907 11691 tablet_service.cc:1514] Processing DeleteTablet for tablet fa0d5100cdbd4de1a88fca7b866557ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:24:25 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:49682
I20260620 20:24:25.950145 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 11623
I20260620 20:24:25.972259 12080 tablet_replica.cc:330] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e: stopping tablet replica
I20260620 20:24:25.973124 12080 raft_consensus.cc:2229] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:24:25.974030 12080 raft_consensus.cc:2258] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:24:25.977025 12080 ts_tablet_manager.cc:1905] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:24:25.980118 12079 tablet_replica.cc:330] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf: stopping tablet replica
I20260620 20:24:25.980448 12078 tablet_replica.cc:330] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d: stopping tablet replica
I20260620 20:24:25.980654 12079 raft_consensus.cc:2229] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:24:25.980825 12078 raft_consensus.cc:2229] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:24:25.981171 12079 raft_consensus.cc:2258] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:24:25.981483 12078 raft_consensus.cc:2258] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:24:25.986842 12080 ts_tablet_manager.cc:1918] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:24:25.987010 12080 log.cc:1196] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/fa0d5100cdbd4de1a88fca7b866557ab
I20260620 20:24:25.987468 12080 ts_tablet_manager.cc:1939] T fa0d5100cdbd4de1a88fca7b866557ab P 4d3978a5769f47c390980f92ec17ee7e: Deleting consensus metadata
I20260620 20:24:25.989207 11529 catalog_manager.cc:4841] TS 4d3978a5769f47c390980f92ec17ee7e (127.31.235.129:36669): tablet fa0d5100cdbd4de1a88fca7b866557ab (table test-table [id=c29eb9841a0b4b778fc616f33c35cfad]) successfully deleted
Jun 20 20:24:25 dist-test-slave-zzwf krb5kdc[11067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:24:25.989964 12079 ts_tablet_manager.cc:1905] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:24:25.991313 12078 ts_tablet_manager.cc:1905] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:24:25.999469 12078 ts_tablet_manager.cc:1918] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:24:25.999643 12078 log.cc:1196] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/fa0d5100cdbd4de1a88fca7b866557ab
I20260620 20:24:26.000157 12078 ts_tablet_manager.cc:1939] T fa0d5100cdbd4de1a88fca7b866557ab P b7232edb1924496b97a36e2e1b993a0d: Deleting consensus metadata
I20260620 20:24:26.001757 12051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:25.951770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49696 (local address 127.31.235.129:36669)
0620 20:24:25.952249 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 20:24:25.952259 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:24:25.971581 (+ 19322us) server_negotiation.cc:407] Connection header received
0620 20:24:25.972552 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:24:25.972558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:24:25.972645 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:24:25.972721 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:24:25.976455 (+ 3734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.977618 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.982194 (+ 4576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:25.982861 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:25.982989 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:25.993985 (+ 10996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:24:25.994001 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:24:25.994018 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:24:25.994069 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:24:25.997903 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:24:25.998909 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:24:25.998932 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:24:25.998936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:24:25.999105 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:24:25.999650 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:24:25.999656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:24:25.999659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:24:26.000174 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:24:26.000532 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0620 20:24:26.001107 (+ 575us) server_negotiation.cc:299] Negotiation successful
0620 20:24:26.001340 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313}
I20260620 20:24:26.001936 11530 catalog_manager.cc:4841] TS b7232edb1924496b97a36e2e1b993a0d (127.31.235.130:35419): tablet fa0d5100cdbd4de1a88fca7b866557ab (table test-table [id=c29eb9841a0b4b778fc616f33c35cfad]) successfully deleted
I20260620 20:24:26.002568 12079 ts_tablet_manager.cc:1918] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:24:26.002736 12079 log.cc:1196] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/fa0d5100cdbd4de1a88fca7b866557ab
I20260620 20:24:26.003235 12079 ts_tablet_manager.cc:1939] T fa0d5100cdbd4de1a88fca7b866557ab P 6308757430844b739f525db7b569f9bf: Deleting consensus metadata
I20260620 20:24:26.003575 11751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:24:26.004959 11527 catalog_manager.cc:4841] TS 6308757430844b739f525db7b569f9bf (127.31.235.131:39813): tablet fa0d5100cdbd4de1a88fca7b866557ab (table test-table [id=c29eb9841a0b4b778fc616f33c35cfad]) successfully deleted
I20260620 20:24:26.097906 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 11623
I20260620 20:24:26.122870 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 11762
Jun 20 20:24:26 dist-test-slave-zzwf krb5kdc[11067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.130@KRBTEST.COM
I20260620 20:24:26.140247 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:26.123737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53486 (local address 127.31.235.130:35419)
0620 20:24:26.123949 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 20:24:26.123958 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:24:26.124035 (+ 77us) server_negotiation.cc:407] Connection header received
0620 20:24:26.124241 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:24:26.124247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:24:26.124324 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:24:26.124432 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:24:26.125781 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:26.126566 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:26.127826 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:26.128248 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:26.128398 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:26.133189 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:24:26.133211 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:24:26.133228 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:24:26.133266 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:24:26.137072 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:24:26.138044 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:24:26.138052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:24:26.138055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:24:26.138209 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:24:26.138619 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:24:26.138628 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:24:26.138631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:24:26.139030 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:24:26.139357 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 20:24:26.139661 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 20:24:26.139910 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:24:26.140954 11894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:24:26.262481 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 11762
I20260620 20:24:26.281062 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 11905
Jun 20 20:24:26 dist-test-slave-zzwf krb5kdc[11067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.131@KRBTEST.COM
I20260620 20:24:26.296614 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:24:26.281814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38654 (local address 127.31.235.131:39813)
0620 20:24:26.282003 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 20:24:26.282007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:24:26.282025 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:24:26.282170 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:24:26.282174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:24:26.282235 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:24:26.282329 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:24:26.283592 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:26.284280 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:26.285666 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:24:26.286125 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:24:26.286233 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:24:26.290840 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:24:26.290857 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:24:26.290874 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:24:26.290915 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:24:26.293966 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:24:26.294741 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:24:26.294749 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:24:26.294752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:24:26.294915 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:24:26.295252 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:24:26.295255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:24:26.295257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:24:26.295607 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:24:26.295832 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0620 20:24:26.296166 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 20:24:26.296340 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:24:26.297255 12028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:24:26.386771 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 11905
I20260620 20:24:26.403545 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 11511
I20260620 20:24:26.404937 11573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:24:26.517292 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 11511
I20260620 20:24:26.538889 32686 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 20:24:26.874207 32686 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 20:24:26.874466 32686 mini_ranger.cc:66] Stopping Ranger...
I20260620 20:24:27.220217 32686 mini_ranger.cc:68] Stopped Ranger
2026-06-20 20:24:27.220 UTC [11097] LOG: received smart shutdown request
2026-06-20 20:24:27.224 UTC [11097] LOG: background worker "logical replication launcher" (PID 11106) exited with exit code 1
2026-06-20 20:24:27.224 UTC [11101] LOG: shutting down
2026-06-20 20:24:27.414 UTC [11097] LOG: database system is shut down
2026-06-20T20:24:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49584 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:24:27 dist-test-slave-zzwf krb5kdc[12094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:24:27 dist-test-slave-zzwf krb5kdc[12094](info): set up 2 sockets
Jun 20 20:24:27 dist-test-slave-zzwf krb5kdc[12094](info): commencing operation
krb5kdc: starting...
W20260620 20:24:28.698267 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:24:28 dist-test-slave-zzwf krb5kdc[12094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:24:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:24:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.235.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.235.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.235.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/rangeradmin_127.31.235.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.235.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/rangeradmin_127.31.235.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.235.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.235.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.235.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/rangerlookup_127.31.235.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.235.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/rangerlookup_127.31.235.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.235.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.235.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.235.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/HTTP_127.31.235.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.235.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/HTTP_127.31.235.148@KRBTEST.COM.keytab.
I20260620 20:24:29.024776 32686 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskgJsYo5/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/postgres -l logfile start
2026-06-20 20:24:31.663 UTC [12124] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 20:24:31.663 UTC [12124] LOG: listening on IPv4 address "127.31.235.148", port 46473
2026-06-20 20:24:31.667 UTC [12124] LOG: listening on Unix socket "/tmp/.s.PGSQL.46473"
2026-06-20 20:24:31.689 UTC [12127] LOG: database system was shut down at 2026-06-20 20:24:30 UTC
2026-06-20 20:24:31.693 UTC [12124] LOG: database system is ready to accept connections
I20260620 20:24:32.673908 32686 mini_postgres.cc:96] Postgres bound to 46473
2026-06-20 20:24:32.690 UTC [12135] FATAL: database "slave" does not exist
127.31.235.148:46473 - accepting connections
I20260620 20:24:32.691587 32686 mini_ranger.cc:161] Starting Ranger...
I20260620 20:24:32.717895 32686 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 20:24:33.247641 32686 mini_ranger.cc:87] Created ranger Postgres database
I20260620 20:24:33.247754 32686 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ranger-admin
2026-06-20 20:24:33,544 [I] DB FLAVOR :POSTGRES
2026-06-20 20:24:33,545 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 20:24:33,545 [I] Checking connection..
2026-06-20 20:24:33,545 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 20:24:33,803 [I] Checking connection passed.
2026-06-20 20:24:33,803 [I] --------- Verifying version history table ---------
2026-06-20 20:24:33,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:24:34,052 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 20:24:34,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:24:34,314 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 20:24:34,314 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 20:24:34,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 20:24:34.556 UTC [12199] WARNING: there is no transaction in progress
2026-06-20 20:24:34,563 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 20:24:34,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 20:24:34,825 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 20:24:34,826 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 20:24:34,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 20:24:35,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 20:24:35,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zzwf.c.gcp-upstream.internal','N') ;"
2026-06-20 20:24:35,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 20:24:35,811 [I] Table x_portal_user does not exist in database ranger
2026-06-20 20:24:35,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 20:24:36,076 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 20:24:36,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 20:24:36,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 20:24:36,581 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 20:24:36,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 20:24:37.562 UTC [12307] WARNING: there is no transaction in progress
2026-06-20 20:24:37.579 UTC [12307] WARNING: there is no transaction in progress
2026-06-20 20:24:37.596 UTC [12307] WARNING: there is no transaction in progress
2026-06-20 20:24:37.611 UTC [12307] WARNING: there is no transaction in progress
2026-06-20 20:24:37.628 UTC [12307] WARNING: there is no transaction in progress
2026-06-20 20:24:37.648 UTC [12307] WARNING: there is no transaction in progress
2026-06-20 20:24:37.713 UTC [12307] WARNING: there is no transaction in progress
2026-06-20 20:24:37.720 UTC [12307] WARNING: there is no transaction in progress
2026-06-20 20:24:37.736 UTC [12307] WARNING: there is no transaction in progress
2026-06-20 20:24:37.746 UTC [12307] WARNING: there is no transaction in progress
2026-06-20 20:24:37.756 UTC [12307] WARNING: there is no transaction in progress
2026-06-20 20:24:37.770 UTC [12307] WARNING: there is no transaction in progress
2026-06-20 20:24:37.780 UTC [12307] WARNING: there is no transaction in progress
2026-06-20 20:24:37.787 UTC [12307] WARNING: there is no transaction in progress
2026-06-20 20:24:37.794 UTC [12307] WARNING: there is no transaction in progress
2026-06-20 20:24:37.802 UTC [12307] WARNING: there is no transaction in progress
2026-06-20 20:24:38,475 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 20:24:38,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 20:24:38,730 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 20:24:38,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 20:24:39,016 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 20:24:39,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 20:24:39,284 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 20:24:39,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 20:24:39,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 20:24:39,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zzwf.c.gcp-upstream.internal';"
2026-06-20 20:24:40,045 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 20:24:40,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 20:24:40,308 [I] DB_PATCHES have already been applied
I20260620 20:24:40.314749 32686 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ranger-admin:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskgJsYo5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 20:24:40.314837 32686 mini_ranger.cc:193] Using host: 127.31.235.148
I20260620 20:24:40.329648 32686 mini_ranger.cc:227] Ranger admin URL: http://127.31.235.148:37427
Jun 20, 2026 8:24:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:24:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:24:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 8:24:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 8:24:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 8:24:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 8:24:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 20:24:41.467012 32686 mini_ranger.cc:160] Time spent starting Ranger: real 8.775s user 0.000s sys 0.039s
Jun 20, 2026 8:25:01 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 8:25:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 8:25:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 8:25:02 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 8:25:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 8:25:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 8:25:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 8:25:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 8:25:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 8:25:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 8:25:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 8:25:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 8:25:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 8:25:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 8:25:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 8:25:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 8:25:02 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 8:25:02 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 20:25:03.382411 32686 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.235.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.235.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.235.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/rangerkms_127.31.235.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.235.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/rangerkms_127.31.235.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.235.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.235.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.235.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/HTTP_127.31.235.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.235.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/HTTP_127.31.235.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 20:25:03 dist-test-slave-zzwf krb5kdc[12094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987103, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 20:25:03.544309 32686 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 20:25:03.564599 32686 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 20:25:04.519318 32686 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 20:25:04.519452 32686 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ranger-kms
I20260620 20:25:04.519508 32686 mini_ranger_kms.cc:227] Using postgres at 127.31.235.148:46473
2026-06-20 20:25:04,873 [I] DB FLAVOR :POSTGRES
2026-06-20 20:25:04,874 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 20:25:04,874 [I] Checking connection
2026-06-20 20:25:04,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 20:25:05,120 [I] connection success
2026-06-20 20:25:05,120 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 20:25:05,121 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 20:25:05,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 20:25:05,391 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 20:25:05,391 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 20:25:05,392 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 20:25:05,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.235.148:46473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 20:25:05,671 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 20:25:06.167088 32686 mini_ranger_kms.cc:310] Created kms service
I20260620 20:25:06.376492 32686 mini_ranger_kms.cc:326] Created kudu user
I20260620 20:25:06.420781 32686 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 20:25:06 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987103, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
I20260620 20:25:06.837332 32686 mini_ranger_kms.cc:384] Added ranger policy
I20260620 20:25:06.837699 32686 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ranger-kms:/tmp/dist-test-taskgJsYo5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskgJsYo5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 20:25:06.837790 32686 mini_ranger_kms.cc:240] Using host: 127.31.235.148
I20260620 20:25:06.852288 32686 mini_ranger_kms.cc:276] Ranger KMS PID: 12487
I20260620 20:25:06.852420 32686 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.235.148:59485
Jun 20, 2026 8:25:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 8:25:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 8:25:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 8:25:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 8:25:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 8:25:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 20:25:09.136202 32686 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.592s user 0.037s sys 0.329s
I20260620 20:25:09.136466 32686 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 20:25:09.136631 32686 mini_ranger_kms.cc:401] 127.31.235.148:59485/kms/v1/keys
Jun 20 20:25:09 dist-test-slave-zzwf krb5kdc[12094](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781987109, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.235.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 20:25:09 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987109, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.235.148@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
Jun 20 20:25:10 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987109, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.235.148@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
Jun 20 20:25:12 dist-test-slave-zzwf krb5kdc[12094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:25:12.349174 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:42485
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:42485
--ranger_config_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.235.148:59485/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:25:12.680707 12538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:12.681000 12538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:12.681171 12538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:12.688686 12538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:25:12.688804 12538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:12.688885 12538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:12.688987 12538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:25:12.689054 12538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:25:12.699458 12538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.235.148:59485/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:42485
--ranger_config_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:42485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:12.700826 12538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:12.702244 12538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:12.715668 12543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:12.715618 12544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:12.716063 12546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:12.717427 12538 server_base.cc:1034] running on GCE node
I20260620 20:25:12.718247 12538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:12.719771 12538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:12.721001 12538 hybrid_clock.cc:648] HybridClock initialized: now 1781987112720953 us; error 39 us; skew 500 ppm
I20260620 20:25:12.721364 12538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:12 dist-test-slave-zzwf krb5kdc[12094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987112, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:12.724678 12538 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:25:12.726387 12538 webserver.cc:460] Webserver started at http://127.31.235.190:37683/ using document root <none> and password file <none>
I20260620 20:25:12.727056 12538 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:12.727193 12538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:12.727505 12538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:25:12 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987112, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
I20260620 20:25:12.776031 12538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "0058323f253a445daf2ea4d1a05fd9e1"
format_stamp: "Formatted at 2026-06-20 20:25:12 on dist-test-slave-zzwf"
server_key: "09d56f658e227b64923951cdc5374cf0"
server_key_iv: "46625c77517df6536280cc86df74dbd8"
server_key_version: "kuduclusterkey@0"
I20260620 20:25:12.777163 12538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "0058323f253a445daf2ea4d1a05fd9e1"
format_stamp: "Formatted at 2026-06-20 20:25:12 on dist-test-slave-zzwf"
server_key: "09d56f658e227b64923951cdc5374cf0"
server_key_iv: "46625c77517df6536280cc86df74dbd8"
server_key_version: "kuduclusterkey@0"
I20260620 20:25:12.784142 12538 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 20:25:12.801527 12553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:12.803033 12538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:25:12.803259 12538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "0058323f253a445daf2ea4d1a05fd9e1"
format_stamp: "Formatted at 2026-06-20 20:25:12 on dist-test-slave-zzwf"
server_key: "09d56f658e227b64923951cdc5374cf0"
server_key_iv: "46625c77517df6536280cc86df74dbd8"
server_key_version: "kuduclusterkey@0"
I20260620 20:25:12.803495 12538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:12.828584 12538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:12.832428 12538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:12.832753 12538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:12.851421 12538 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:42485
I20260620 20:25:12.851441 12604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:42485 every 8 connection(s)
I20260620 20:25:12.852761 12538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:25:12.856519 12605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:25:12.862622 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 12538
I20260620 20:25:12.862833 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:25:12.863471 32686 external_mini_cluster.cc:1428] Setting key 23ff454fa408514eb8137be7ef1d66da
I20260620 20:25:12.868158 12605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1: Bootstrap starting.
I20260620 20:25:12.872087 12605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1: Neither blocks nor log segments found. Creating new log.
I20260620 20:25:12.873348 12605 log.cc:824] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1: Log is configured to *not* fsync() on all Append() calls
I20260620 20:25:12.876742 12605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1: No bootstrap required, opened a new log
Jun 20 20:25:12 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:25:12.886718 12605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0058323f253a445daf2ea4d1a05fd9e1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42485 } }
I20260620 20:25:12.887164 12605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:25:12.887279 12605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0058323f253a445daf2ea4d1a05fd9e1, State: Initialized, Role: FOLLOWER
I20260620 20:25:12.887957 12605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0058323f253a445daf2ea4d1a05fd9e1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42485 } }
I20260620 20:25:12.888211 12608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:12.865608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42498 (local address 127.31.235.190:42485)
0620 20:25:12.866401 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:12.866417 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:12.866467 (+ 50us) server_negotiation.cc:407] Connection header received
0620 20:25:12.868827 (+ 2360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:12.868856 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:12.869335 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:12.869791 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:12.871008 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:12.873004 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:12.874767 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:12.875777 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:12.875958 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:12.879775 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:12.879786 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:12.879806 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:12.879848 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:12.883857 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:12.884484 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:12.884497 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:12.884508 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:12.884707 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:12.885061 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:12.885066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:12.885070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:12.885833 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:12.886109 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:12.886485 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 20:25:12.886879 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":320,"threads_started":1}
I20260620 20:25:12.888234 12605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:25:12.888392 12605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:25:12.888576 12605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:25:12.891261 12605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0058323f253a445daf2ea4d1a05fd9e1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42485 } }
I20260620 20:25:12.891688 12605 leader_election.cc:302] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0058323f253a445daf2ea4d1a05fd9e1; no voters:
I20260620 20:25:12.892227 12605 leader_election.cc:288] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:25:12.892390 12610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:25:12.893074 12610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [term 1 LEADER]: Becoming Leader. State: Replica: 0058323f253a445daf2ea4d1a05fd9e1, State: Running, Role: LEADER
I20260620 20:25:12.893731 12610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0058323f253a445daf2ea4d1a05fd9e1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42485 } }
I20260620 20:25:12.894305 12605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:25:12.896546 12612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0058323f253a445daf2ea4d1a05fd9e1. Latest consensus state: current_term: 1 leader_uuid: "0058323f253a445daf2ea4d1a05fd9e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0058323f253a445daf2ea4d1a05fd9e1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42485 } } }
I20260620 20:25:12.896970 12612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:25:12.897801 12611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0058323f253a445daf2ea4d1a05fd9e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0058323f253a445daf2ea4d1a05fd9e1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42485 } } }
I20260620 20:25:12.898316 12611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:25:12.902501 12615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:25:12.907752 12615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:25:12.908546 12605 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 20:25:12.924980 12615 catalog_manager.cc:1347] Generated new cluster ID: a9fce49a8f684e03bcae003a212bec53
I20260620 20:25:12.925169 12615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:25:12.939750 12615 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:25:12.941803 12615 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:25:12.957682 12615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1: Generated new TSK 0
I20260620 20:25:12.959359 12615 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
I20260620 20:25:13.723774 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.235.148:59485/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:42485
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:25:14 dist-test-slave-zzwf krb5kdc[12094](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781987114, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 20:25:14.152130 12650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:14.152462 12650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:14.152670 12650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:14.160830 12650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:14.160977 12650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:14.161171 12650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:25:14.171259 12650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.235.148:59485/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:42485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:14.172739 12650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:14.174252 12650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:14.193949 12661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:14.195040 12660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:14.195498 12650 server_base.cc:1034] running on GCE node
W20260620 20:25:14.195899 12663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:14.196698 12650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:14.203310 12650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:14.204643 12650 hybrid_clock.cc:648] HybridClock initialized: now 1781987114204569 us; error 68 us; skew 500 ppm
I20260620 20:25:14.205077 12650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:14 dist-test-slave-zzwf krb5kdc[12094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987114, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:14.209430 12650 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:25:14.213191 12650 webserver.cc:460] Webserver started at http://127.31.235.129:44047/ using document root <none> and password file <none>
I20260620 20:25:14.214198 12650 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:14.214411 12650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:14.214839 12650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:25:14 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987114, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
I20260620 20:25:14.239125 12650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "f7a27a55420c4407a6613a3b063e1e1a"
format_stamp: "Formatted at 2026-06-20 20:25:14 on dist-test-slave-zzwf"
server_key: "4092a9e5106e3f82c5d31329530e2ce6"
server_key_iv: "5971151497b8449f3b8400d5f80c300a"
server_key_version: "kuduclusterkey@0"
I20260620 20:25:14.240293 12650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "f7a27a55420c4407a6613a3b063e1e1a"
format_stamp: "Formatted at 2026-06-20 20:25:14 on dist-test-slave-zzwf"
server_key: "4092a9e5106e3f82c5d31329530e2ce6"
server_key_iv: "5971151497b8449f3b8400d5f80c300a"
server_key_version: "kuduclusterkey@0"
I20260620 20:25:14.250855 12650 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.004s
I20260620 20:25:14.290249 12670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:14.292363 12650 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.004s sys 0.012s
I20260620 20:25:14.292632 12650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "f7a27a55420c4407a6613a3b063e1e1a"
format_stamp: "Formatted at 2026-06-20 20:25:14 on dist-test-slave-zzwf"
server_key: "4092a9e5106e3f82c5d31329530e2ce6"
server_key_iv: "5971151497b8449f3b8400d5f80c300a"
server_key_version: "kuduclusterkey@0"
I20260620 20:25:14.292889 12650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:14.318267 12650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:14.323365 12650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:14.323702 12650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:14.324877 12650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:25:14.326658 12650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:25:14.326749 12650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:14.326859 12650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:25:14.326917 12650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:14.440469 12650 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:35491
I20260620 20:25:14.440485 12783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:35491 every 8 connection(s)
I20260620 20:25:14.442593 12650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:25:14.451320 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 12650
I20260620 20:25:14.451545 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:25:14.452382 32686 external_mini_cluster.cc:1428] Setting key 6ab883cf3a4415a8eff93903792406cc
Jun 20 20:25:14 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987114, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.130@KRBTEST.COM" created.
I20260620 20:25:14.494936 12786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:14.446394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:53807 (local address 127.31.235.190:42485)
0620 20:25:14.462436 (+ 16042us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:14.462444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:14.462471 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:25:14.462546 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:14.462554 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:14.462661 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:14.465290 (+ 2629us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:14.465516 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:14.466792 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:14.476527 (+ 9735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:14.477284 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:14.477432 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:14.482771 (+ 5339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:14.482778 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:14.482783 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:14.482820 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:14.486269 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:14.487278 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:14.487283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:14.487287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:14.487456 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:14.487988 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:14.487992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:14.487996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:14.492518 (+ 4522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:14.494095 (+ 1577us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:14.494234 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 20:25:14.494524 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15876,"thread_start_us":16374,"threads_started":1}
I20260620 20:25:14.499292 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:14.446922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:42485 (local address 127.31.235.129:53807)
0620 20:25:14.450973 (+ 4051us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:14.451038 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:14.456605 (+ 5567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:14.463284 (+ 6679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:14.463301 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:14.464056 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:14.465060 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:14.465090 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:14.473692 (+ 8602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:14.473702 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:14.476248 (+ 2546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:14.476275 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:14.477707 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:14.477719 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:14.478143 (+ 424us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:14.479108 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:14.479145 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:14.482503 (+ 3358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:14.486448 (+ 3945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:14.486458 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:14.486478 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:14.487098 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:14.487599 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:14.487605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:14.487609 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:14.487851 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:14.492675 (+ 4824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:14.492685 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:14.493148 (+ 463us) client_negotiation.cc:769] Sending connection context
0620 20:25:14.494459 (+ 1311us) client_negotiation.cc:240] Negotiation successful
0620 20:25:14.497584 (+ 3125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3655,"thread_start_us":3379,"threads_started":1}
I20260620 20:25:14.501682 12784 heartbeater.cc:347] Connected to a master server at 127.31.235.190:42485
I20260620 20:25:14.502002 12784 heartbeater.cc:464] Registering TS with master...
I20260620 20:25:14.502828 12784 heartbeater.cc:511] Master 127.31.235.190:42485 requested a full tablet report, sending...
I20260620 20:25:14.505460 12570 ts_manager.cc:194] Registered new tserver with Master: f7a27a55420c4407a6613a3b063e1e1a (127.31.235.129:35491)
I20260620 20:25:14.507831 12570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:53807
Jun 20 20:25:14 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987114, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
I20260620 20:25:14.525549 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.235.148:59485/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:42485
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:25:14 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987114, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
W20260620 20:25:14.996984 12790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:14.997447 12790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:14.997653 12790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:15.008997 12790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:15.009202 12790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:15.009411 12790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:25:15.024761 12790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.235.148:59485/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:42485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:15.026587 12790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:15.028205 12790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:15.044801 12797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:15.060854 12798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:15.061461 12800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:15.063122 12790 server_base.cc:1034] running on GCE node
I20260620 20:25:15.063925 12790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:15.065188 12790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:15.066502 12790 hybrid_clock.cc:648] HybridClock initialized: now 1781987115066375 us; error 113 us; skew 500 ppm
I20260620 20:25:15.066869 12790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:15 dist-test-slave-zzwf krb5kdc[12094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987115, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:15.070675 12790 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:25:15.072373 12790 webserver.cc:460] Webserver started at http://127.31.235.130:36779/ using document root <none> and password file <none>
I20260620 20:25:15.073127 12790 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:15.073271 12790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:15.073676 12790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:25:15 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987115, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
I20260620 20:25:15.090097 12790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "8b09f8b27d46413586e1387dc3c849ef"
format_stamp: "Formatted at 2026-06-20 20:25:15 on dist-test-slave-zzwf"
server_key: "8828172e599b5e85f936eb52c97cb50d"
server_key_iv: "55d3e73d11da1ba4fbb600c1f58aaa6e"
server_key_version: "kuduclusterkey@0"
I20260620 20:25:15.091276 12790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "8b09f8b27d46413586e1387dc3c849ef"
format_stamp: "Formatted at 2026-06-20 20:25:15 on dist-test-slave-zzwf"
server_key: "8828172e599b5e85f936eb52c97cb50d"
server_key_iv: "55d3e73d11da1ba4fbb600c1f58aaa6e"
server_key_version: "kuduclusterkey@0"
I20260620 20:25:15.097359 12790 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 20:25:15.115438 12814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:15.116992 12790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:25:15.117198 12790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "8b09f8b27d46413586e1387dc3c849ef"
format_stamp: "Formatted at 2026-06-20 20:25:15 on dist-test-slave-zzwf"
server_key: "8828172e599b5e85f936eb52c97cb50d"
server_key_iv: "55d3e73d11da1ba4fbb600c1f58aaa6e"
server_key_version: "kuduclusterkey@0"
I20260620 20:25:15.117434 12790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:15.135308 12790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:15.139557 12790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:15.139879 12790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:15.140913 12790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:25:15.142340 12790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:25:15.142411 12790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:15.142542 12790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:25:15.142596 12790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:15.180578 12790 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:40263
I20260620 20:25:15.180615 12927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:40263 every 8 connection(s)
I20260620 20:25:15.182128 12790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:25:15.186678 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 12790
I20260620 20:25:15.186843 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:25:15.187408 32686 external_mini_cluster.cc:1428] Setting key a2023d0473b174afd31cc178e3569f27
Jun 20 20:25:15 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987115, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:25:15.210126 12930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:15.185609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:52553 (local address 127.31.235.190:42485)
0620 20:25:15.186164 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:15.186170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:15.187582 (+ 1412us) server_negotiation.cc:407] Connection header received
0620 20:25:15.189957 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:15.189964 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:15.190052 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:15.190167 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:15.192208 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:15.193640 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:15.196167 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:15.196908 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:15.197059 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:15.201877 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:15.201884 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:15.201889 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:15.201932 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:15.205630 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:15.206766 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:15.206772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:15.206776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:15.206935 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:15.207607 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:15.207613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:15.207616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:15.208012 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:15.208199 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:15.209282 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0620 20:25:15.209604 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":269,"threads_started":1}
I20260620 20:25:15.211146 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:15.186172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:42485 (local address 127.31.235.130:52553)
0620 20:25:15.187143 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:15.187198 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:15.189627 (+ 2429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:15.190452 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:15.190466 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:15.191093 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:15.191982 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:15.192011 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:15.193843 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:15.193849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:15.195970 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:15.195988 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:15.197188 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:15.197197 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:15.197541 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:15.198443 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:15.198475 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:15.201585 (+ 3110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:15.205807 (+ 4222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:15.205820 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:15.205842 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:15.206620 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:15.207068 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:15.207076 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:15.207083 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:15.207384 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:15.208278 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:15.208287 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:15.208740 (+ 453us) client_negotiation.cc:769] Sending connection context
0620 20:25:15.209135 (+ 395us) client_negotiation.cc:240] Negotiation successful
0620 20:25:15.209741 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":845,"threads_started":1}
WARNING: no policy specified for kudu/127.31.235.131@KRBTEST.COM; defaulting to no policy
I20260620 20:25:15.212735 12928 heartbeater.cc:347] Connected to a master server at 127.31.235.190:42485
I20260620 20:25:15.213056 12928 heartbeater.cc:464] Registering TS with master...
I20260620 20:25:15.213953 12928 heartbeater.cc:511] Master 127.31.235.190:42485 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.131@KRBTEST.COM" created.
I20260620 20:25:15.215760 12570 ts_manager.cc:194] Registered new tserver with Master: 8b09f8b27d46413586e1387dc3c849ef (127.31.235.130:40263)
I20260620 20:25:15.216579 12570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:52553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
I20260620 20:25:15.245922 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.235.148:59485/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:42485
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:25:15.511680 12784 heartbeater.cc:503] Master 127.31.235.190:42485 was elected leader, sending a full tablet report...
W20260620 20:25:15.583804 12935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:15.584141 12935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:15.584298 12935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:15.591899 12935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:15.592057 12935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:15.592221 12935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:25:15.602375 12935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.235.148:59485/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:42485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:15.603718 12935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:15.604995 12935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:15.618065 12940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:15.618034 12943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:15.618129 12941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:15.618901 12935 server_base.cc:1034] running on GCE node
I20260620 20:25:15.619445 12935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:15.620404 12935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:15.621635 12935 hybrid_clock.cc:648] HybridClock initialized: now 1781987115621561 us; error 60 us; skew 500 ppm
I20260620 20:25:15.621991 12935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:15 dist-test-slave-zzwf krb5kdc[12094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987115, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:15.625104 12935 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:25:15.626608 12935 webserver.cc:460] Webserver started at http://127.31.235.131:44363/ using document root <none> and password file <none>
I20260620 20:25:15.627266 12935 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:15.627365 12935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:15.627609 12935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 20:25:15 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987115, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
I20260620 20:25:15.644420 12935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "ea7eb53221df4d3db69fdd03940e9f3f"
format_stamp: "Formatted at 2026-06-20 20:25:15 on dist-test-slave-zzwf"
server_key: "c9aaf74165bc4f1345b451f1eb13479d"
server_key_iv: "778eb4d414892d266199b5dd7fe57f75"
server_key_version: "kuduclusterkey@0"
I20260620 20:25:15.645254 12935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "ea7eb53221df4d3db69fdd03940e9f3f"
format_stamp: "Formatted at 2026-06-20 20:25:15 on dist-test-slave-zzwf"
server_key: "c9aaf74165bc4f1345b451f1eb13479d"
server_key_iv: "778eb4d414892d266199b5dd7fe57f75"
server_key_version: "kuduclusterkey@0"
I20260620 20:25:15.650494 12935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 20:25:15.666464 12950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:15.668040 12935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:25:15.668206 12935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "ea7eb53221df4d3db69fdd03940e9f3f"
format_stamp: "Formatted at 2026-06-20 20:25:15 on dist-test-slave-zzwf"
server_key: "c9aaf74165bc4f1345b451f1eb13479d"
server_key_iv: "778eb4d414892d266199b5dd7fe57f75"
server_key_version: "kuduclusterkey@0"
I20260620 20:25:15.668440 12935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:15.695087 12935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:15.699229 12935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:15.699523 12935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:15.700533 12935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:25:15.702000 12935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:25:15.702072 12935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:15.702193 12935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:25:15.702248 12935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:15.735088 12935 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:38909
I20260620 20:25:15.735131 13063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:38909 every 8 connection(s)
I20260620 20:25:15.736393 12935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:25:15.745029 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 12935
I20260620 20:25:15.745225 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:25:15.745751 32686 external_mini_cluster.cc:1428] Setting key e380dd6b4f9665396f9e7bdbc1396db7
Jun 20 20:25:15 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987115, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:25:15.755268 13066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:15.738886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:43143 (local address 127.31.235.190:42485)
0620 20:25:15.739412 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:15.739419 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:15.740230 (+ 811us) server_negotiation.cc:407] Connection header received
0620 20:25:15.741989 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:15.741994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:15.742062 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:15.742157 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:15.743433 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:15.744142 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:15.745856 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:15.746422 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:15.746564 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:15.750333 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:15.750338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:15.750341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:15.750371 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:15.752697 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:15.753321 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:15.753325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:15.753327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:15.753444 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:15.753892 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:15.753895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:15.753897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:15.754156 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:15.754273 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:15.754847 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 20:25:15.755025 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":230,"threads_started":1}
I20260620 20:25:15.756193 13067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:15.739360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:42485 (local address 127.31.235.131:43143)
0620 20:25:15.740082 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:15.740123 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:15.741741 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:15.742346 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:15.742354 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:15.742753 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:15.743284 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:15.743303 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:15.744267 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:15.744271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:15.745718 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:15.745730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:15.746654 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:15.746659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:15.746931 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:15.747769 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:15.747802 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:15.750142 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:15.752823 (+ 2681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:15.752829 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:15.752840 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:15.753211 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:15.753615 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:15.753619 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:15.753621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:15.753795 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:15.754255 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:15.754260 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:15.754525 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 20:25:15.754758 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 20:25:15.755094 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":268,"threads_started":1}
I20260620 20:25:15.757678 13064 heartbeater.cc:347] Connected to a master server at 127.31.235.190:42485
I20260620 20:25:15.757966 13064 heartbeater.cc:464] Registering TS with master...
I20260620 20:25:15.758627 13064 heartbeater.cc:511] Master 127.31.235.190:42485 requested a full tablet report, sending...
I20260620 20:25:15.760143 12570 ts_manager.cc:194] Registered new tserver with Master: ea7eb53221df4d3db69fdd03940e9f3f (127.31.235.131:38909)
I20260620 20:25:15.761123 12570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:43143
I20260620 20:25:15.761763 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:25:15.764112 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 12650
Jun 20 20:25:15 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:25:15.780486 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:15.765099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59054 (local address 127.31.235.129:35491)
0620 20:25:15.765751 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:15.765757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:15.765783 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:25:15.765846 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:15.765852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:15.766045 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:15.766179 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:15.767569 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:15.768564 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:15.770149 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:15.770726 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:15.770841 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:15.775075 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:15.775087 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:15.775099 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:15.775130 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:15.777783 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:15.778477 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:15.778484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:15.778486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:15.778620 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:15.778982 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:15.778986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:15.778988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:15.779324 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:15.779567 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:15.779918 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 20:25:15.780200 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":330,"threads_started":1}
I20260620 20:25:15.781754 12778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:15.865370 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 12650
I20260620 20:25:15.881119 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 12790
Jun 20 20:25:15 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.130@KRBTEST.COM
I20260620 20:25:15.897281 13071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:15.882051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40126 (local address 127.31.235.130:40263)
0620 20:25:15.882605 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:15.882612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:15.882637 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:25:15.882695 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:15.882702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:15.882912 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:15.883039 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:15.884208 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:15.885219 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:15.886606 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:15.887191 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:15.887293 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:15.891843 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:15.891855 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:15.891869 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:15.891943 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:15.894616 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:15.895430 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:15.895437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:15.895439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:15.895565 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:15.895881 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:15.895884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:15.895886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:15.896246 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:15.896459 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:15.896685 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 20:25:15.896879 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":289,"threads_started":1}
I20260620 20:25:15.898825 12923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:15.980170 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 12790
I20260620 20:25:15.996021 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 12935
Jun 20 20:25:16 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.131@KRBTEST.COM
I20260620 20:25:16.013010 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:15.997077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47356 (local address 127.31.235.131:38909)
0620 20:25:15.997757 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:15.997764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:15.997794 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:25:15.997861 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:15.997867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:15.998074 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:15.998253 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:15.999546 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:16.000666 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:16.002007 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:16.002551 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:16.002698 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:16.007294 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:16.007305 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:16.007320 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:16.007357 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:16.010314 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:16.010998 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:16.011005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:16.011008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:16.011161 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:16.011536 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:16.011540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:16.011542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:16.011890 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:16.012136 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:16.012392 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 20:25:16.012591 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":307,"threads_started":1}
I20260620 20:25:16.014446 13059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:16.098448 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 12935
I20260620 20:25:16.114230 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 12538
I20260620 20:25:16.115566 12600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:16.223393 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 12538
I20260620 20:25:16.247196 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:42485
--webserver_interface=127.31.235.190
--webserver_port=37683
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:42485
--ranger_config_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.235.148:59485/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.235.148:59485/kms with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:25:16.572613 13076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:16.572883 13076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:16.573081 13076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:16.580314 13076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:25:16.580418 13076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:16.580503 13076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:16.580603 13076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:25:16.580693 13076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:25:16.590837 13076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.235.148:59485/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:42485
--ranger_config_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:42485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=37683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:16.592202 13076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:16.593577 13076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:16.605512 13082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:16.605540 13081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:16.605515 13084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:16.606812 13076 server_base.cc:1034] running on GCE node
I20260620 20:25:16.607649 13076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:16.609124 13076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:16.610365 13076 hybrid_clock.cc:648] HybridClock initialized: now 1781987116610333 us; error 57 us; skew 500 ppm
I20260620 20:25:16.610726 13076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:16 dist-test-slave-zzwf krb5kdc[12094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987116, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:16.613824 13076 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:25:16.615370 13076 webserver.cc:460] Webserver started at http://127.31.235.190:37683/ using document root <none> and password file <none>
I20260620 20:25:16.616055 13076 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:16.616201 13076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 20:25:16 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987116, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
I20260620 20:25:16.674332 13076 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:25:16.678254 13092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:16.680019 13076 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 20:25:16.680220 13076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "0058323f253a445daf2ea4d1a05fd9e1"
format_stamp: "Formatted at 2026-06-20 20:25:12 on dist-test-slave-zzwf"
server_key: "09d56f658e227b64923951cdc5374cf0"
server_key_iv: "46625c77517df6536280cc86df74dbd8"
server_key_version: "kuduclusterkey@0"
I20260620 20:25:16.681569 13076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:16.709128 13076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:16.713521 13076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:16.713771 13076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:16.732826 13076 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:42485
I20260620 20:25:16.732827 13143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:42485 every 8 connection(s)
I20260620 20:25:16.734346 13076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:25:16.740989 13144 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:25:16.743407 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 13076
I20260620 20:25:16.744240 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.235.148:59485/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:35491
--local_ip_for_outbound_sockets=127.31.235.129
--tserver_master_addrs=127.31.235.190:42485
--webserver_port=44047
--webserver_interface=127.31.235.129
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.235.148:59485/kms with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:25:16.744345 13144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1: Bootstrap starting.
I20260620 20:25:16.762780 13144 log.cc:824] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1: Log is configured to *not* fsync() on all Append() calls
I20260620 20:25:16.771175 13144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:25:16.771688 13144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1: Bootstrap complete.
I20260620 20:25:16.780468 13144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0058323f253a445daf2ea4d1a05fd9e1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42485 } }
I20260620 20:25:16.781301 13144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0058323f253a445daf2ea4d1a05fd9e1, State: Initialized, Role: FOLLOWER
I20260620 20:25:16.781929 13144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0058323f253a445daf2ea4d1a05fd9e1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42485 } }
I20260620 20:25:16.782161 13144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:25:16.782291 13144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:25:16.782475 13144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:25:16.786576 13144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0058323f253a445daf2ea4d1a05fd9e1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42485 } }
I20260620 20:25:16.787076 13144 leader_election.cc:302] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0058323f253a445daf2ea4d1a05fd9e1; no voters:
I20260620 20:25:16.787694 13144 leader_election.cc:288] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 20:25:16.787829 13148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:25:16.789995 13144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:25:16.793807 13148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [term 2 LEADER]: Becoming Leader. State: Replica: 0058323f253a445daf2ea4d1a05fd9e1, State: Running, Role: LEADER
I20260620 20:25:16.794622 13148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0058323f253a445daf2ea4d1a05fd9e1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42485 } }
I20260620 20:25:16.797756 13150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0058323f253a445daf2ea4d1a05fd9e1. Latest consensus state: current_term: 2 leader_uuid: "0058323f253a445daf2ea4d1a05fd9e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0058323f253a445daf2ea4d1a05fd9e1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42485 } } }
I20260620 20:25:16.798460 13150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:25:16.798081 13149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0058323f253a445daf2ea4d1a05fd9e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0058323f253a445daf2ea4d1a05fd9e1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42485 } } }
I20260620 20:25:16.799084 13149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:25:16.808545 13157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:25:16.815459 13157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:25:16.816632 13144 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 20:25:16.816982 13157 catalog_manager.cc:1259] Loaded cluster ID: a9fce49a8f684e03bcae003a212bec53
I20260620 20:25:16.817103 13157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:25:16.821988 13157 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:25:16.823860 13157 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1: Loaded TSK: 0
I20260620 20:25:16.825838 13157 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:25:17.209066 13146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:17.209446 13146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:17.209664 13146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:17.221385 13146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:17.221612 13146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:17.221845 13146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:25:17.238077 13146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.235.148:59485/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:35491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=44047
--tserver_master_addrs=127.31.235.190:42485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:17.240031 13146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:17.241888 13146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:17.278944 13181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:17.280711 13182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:17.289186 13146 server_base.cc:1034] running on GCE node
W20260620 20:25:17.290237 13184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:17.291191 13146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:17.298727 13146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:17.302201 13146 hybrid_clock.cc:648] HybridClock initialized: now 1781987117301934 us; error 253 us; skew 500 ppm
I20260620 20:25:17.302968 13146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:17 dist-test-slave-zzwf krb5kdc[12094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987117, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:17.312713 13146 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:25:17.314810 13146 webserver.cc:460] Webserver started at http://127.31.235.129:44047/ using document root <none> and password file <none>
I20260620 20:25:17.315831 13146 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:17.316004 13146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 20:25:17 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987117, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
I20260620 20:25:17.371580 13146 fs_manager.cc:709] Time spent opening directory manager: real 0.027s user 0.006s sys 0.000s
I20260620 20:25:17.386490 13193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:17.388548 13146 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.001s sys 0.011s
I20260620 20:25:17.388808 13146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "f7a27a55420c4407a6613a3b063e1e1a"
format_stamp: "Formatted at 2026-06-20 20:25:14 on dist-test-slave-zzwf"
server_key: "4092a9e5106e3f82c5d31329530e2ce6"
server_key_iv: "5971151497b8449f3b8400d5f80c300a"
server_key_version: "kuduclusterkey@0"
I20260620 20:25:17.390437 13146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:17.428952 13146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:17.433399 13146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:17.433776 13146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:17.435226 13146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:25:17.442206 13146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:25:17.442322 13146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:17.442476 13146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:25:17.442560 13146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:17.496560 13146 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:35491
I20260620 20:25:17.496614 13306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:35491 every 8 connection(s)
I20260620 20:25:17.498169 13146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:25:17.506311 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 13146
I20260620 20:25:17.507063 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.235.148:59485/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:40263
--local_ip_for_outbound_sockets=127.31.235.130
--tserver_master_addrs=127.31.235.190:42485
--webserver_port=36779
--webserver_interface=127.31.235.130
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.235.148:59485/kms with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:25:17 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987117, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:25:17.528153 13309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:17.501351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:42485 (local address 127.31.235.129:56749)
0620 20:25:17.503145 (+ 1794us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:17.503197 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:17.504891 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:17.507265 (+ 2374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:17.507276 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:17.507911 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:17.508681 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:17.508709 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:17.511203 (+ 2494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:17.511209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:17.513177 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:17.513192 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:17.514765 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:17.514773 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:17.515091 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:17.515910 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:17.515941 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:17.518679 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:17.523171 (+ 4492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:17.523181 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:17.523197 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:17.523781 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:17.524300 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:17.524305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:17.524309 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:17.524544 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:17.525613 (+ 1069us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:17.525621 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:17.526036 (+ 415us) client_negotiation.cc:769] Sending connection context
0620 20:25:17.526334 (+ 298us) client_negotiation.cc:240] Negotiation successful
0620 20:25:17.526741 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1542,"thread_start_us":1360,"threads_started":1}
I20260620 20:25:17.528788 13310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:17.501437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:56749 (local address 127.31.235.190:42485)
0620 20:25:17.502553 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:17.502589 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:17.503361 (+ 772us) server_negotiation.cc:407] Connection header received
0620 20:25:17.505610 (+ 2249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:17.505633 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:17.506131 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:17.506572 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:17.508942 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:17.510992 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:17.513374 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:17.514489 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:17.514698 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:17.518982 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:17.518992 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:17.519012 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:17.519057 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:17.522993 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:17.523957 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:17.523966 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:17.523974 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:17.524166 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:17.524688 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:17.524693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:17.524697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:17.525303 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:17.525507 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:17.527257 (+ 1750us) server_negotiation.cc:299] Negotiation successful
0620 20:25:17.527635 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":361,"threads_started":1}
I20260620 20:25:17.532379 13307 heartbeater.cc:347] Connected to a master server at 127.31.235.190:42485
I20260620 20:25:17.532732 13307 heartbeater.cc:464] Registering TS with master...
I20260620 20:25:17.533612 13307 heartbeater.cc:511] Master 127.31.235.190:42485 requested a full tablet report, sending...
W20260620 20:25:17.536674 13307 heartbeater.cc:644] Failed to heartbeat to 127.31.235.190:42485 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 20:25:17.536918 13307 heartbeater.cc:464] Registering TS with master...
I20260620 20:25:17.537222 13307 heartbeater.cc:511] Master 127.31.235.190:42485 requested a full tablet report, sending...
I20260620 20:25:17.538533 13307 heartbeater.cc:464] Registering TS with master...
I20260620 20:25:17.538856 13307 heartbeater.cc:511] Master 127.31.235.190:42485 requested a full tablet report, sending...
W20260620 20:25:17.539959 13307 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:25:17.925954 13311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:17.926405 13311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:17.926563 13311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:17.939633 13311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:17.939795 13311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:17.940057 13311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:25:17.956125 13311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.235.148:59485/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:40263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=36779
--tserver_master_addrs=127.31.235.190:42485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:17.958012 13311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:17.959767 13311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:17.981446 13327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:17.983780 13329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:17.984388 13326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:17.990183 13311 server_base.cc:1034] running on GCE node
I20260620 20:25:17.991161 13311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:18.001847 13311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:18.005319 13311 hybrid_clock.cc:648] HybridClock initialized: now 1781987118005235 us; error 90 us; skew 500 ppm
I20260620 20:25:18.005759 13311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:18 dist-test-slave-zzwf krb5kdc[12094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987118, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:18.009265 13311 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:25:18.011246 13311 webserver.cc:460] Webserver started at http://127.31.235.130:36779/ using document root <none> and password file <none>
I20260620 20:25:18.011991 13311 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:18.012168 13311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 20:25:18 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987118, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
I20260620 20:25:18.036526 13311 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 20:25:18.040212 13338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:18.041814 13311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:25:18.042009 13311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "8b09f8b27d46413586e1387dc3c849ef"
format_stamp: "Formatted at 2026-06-20 20:25:15 on dist-test-slave-zzwf"
server_key: "8828172e599b5e85f936eb52c97cb50d"
server_key_iv: "55d3e73d11da1ba4fbb600c1f58aaa6e"
server_key_version: "kuduclusterkey@0"
I20260620 20:25:18.043119 13311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 20 20:25:18 dist-test-slave-zzwf krb5kdc[12094](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781987118, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:18.062695 13311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:18.067231 13311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:18.067582 13311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:18.068820 13311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:25:18.070363 13311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:25:18.070489 13311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:18.070633 13311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:25:18.070724 13311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:18.143990 13311 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:40263
I20260620 20:25:18.144037 13452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:40263 every 8 connection(s)
I20260620 20:25:18.145601 13311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:25:18.148465 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 13311
I20260620 20:25:18.153364 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.235.148:59485/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:38909
--local_ip_for_outbound_sockets=127.31.235.131
--tserver_master_addrs=127.31.235.190:42485
--webserver_port=44363
--webserver_interface=127.31.235.131
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:34619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.235.148:59485/kms with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:25:18 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987118, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:25:18.199872 13455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:18.159075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:54615 (local address 127.31.235.190:42485)
0620 20:25:18.159605 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:18.159614 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:18.176315 (+ 16701us) server_negotiation.cc:407] Connection header received
0620 20:25:18.180325 (+ 4010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:18.180332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:18.180424 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:18.180509 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:18.183078 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:18.184205 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:18.186951 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:18.187618 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:18.187722 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:18.192271 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:18.192278 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:18.192283 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:18.192323 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:18.195986 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:18.196997 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:18.197003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:18.197007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:18.197158 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:18.197723 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:18.197728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:18.197732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:18.198107 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:18.198241 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:18.199247 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0620 20:25:18.199495 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":272,"threads_started":1}
I20260620 20:25:18.202303 13457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:18.159774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:42485 (local address 127.31.235.130:54615)
0620 20:25:18.176076 (+ 16302us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:18.176157 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:18.179571 (+ 3414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:18.181091 (+ 1520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:18.181104 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:18.181789 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:18.182797 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:18.182828 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:18.184427 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:18.184432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:18.186771 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:18.186787 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:18.188119 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:18.188126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:18.188500 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:18.189376 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:18.189414 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:18.192003 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:18.196169 (+ 4166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:18.196179 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:18.196200 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:18.196840 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:18.197300 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:18.197305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:18.197308 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:18.197592 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:18.198338 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:18.198346 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:18.198797 (+ 451us) client_negotiation.cc:769] Sending connection context
0620 20:25:18.200465 (+ 1668us) client_negotiation.cc:240] Negotiation successful
0620 20:25:18.201017 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15924,"thread_start_us":16445,"threads_started":1}
I20260620 20:25:18.204236 13453 heartbeater.cc:347] Connected to a master server at 127.31.235.190:42485
I20260620 20:25:18.204538 13453 heartbeater.cc:464] Registering TS with master...
I20260620 20:25:18.205376 13453 heartbeater.cc:511] Master 127.31.235.190:42485 requested a full tablet report, sending...
I20260620 20:25:18.207461 13109 ts_manager.cc:194] Registered new tserver with Master: 8b09f8b27d46413586e1387dc3c849ef (127.31.235.130:40263)
I20260620 20:25:18.209605 13109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:54615
Jun 20 20:25:18 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987118, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
I20260620 20:25:18.540127 13307 heartbeater.cc:464] Registering TS with master...
I20260620 20:25:18.540597 13307 heartbeater.cc:511] Master 127.31.235.190:42485 requested a full tablet report, sending...
I20260620 20:25:18.541880 13109 ts_manager.cc:194] Registered new tserver with Master: f7a27a55420c4407a6613a3b063e1e1a (127.31.235.129:35491)
I20260620 20:25:18.543076 13109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:56749
W20260620 20:25:18.667662 13456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:18.668093 13456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:18.668311 13456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:18.678902 13456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:18.679080 13456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:18.679256 13456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:25:18.690378 13456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:34619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.235.148:59485/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:38909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=44363
--tserver_master_addrs=127.31.235.190:42485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:18.691828 13456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:18.693145 13456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:18.711869 13462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:18.717159 13463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:18.727097 13465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:18.727857 13456 server_base.cc:1034] running on GCE node
I20260620 20:25:18.728564 13456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:18.729714 13456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:18.731006 13456 hybrid_clock.cc:648] HybridClock initialized: now 1781987118730945 us; error 57 us; skew 500 ppm
I20260620 20:25:18.731353 13456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:18 dist-test-slave-zzwf krb5kdc[12094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987118, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:18.736377 13456 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:25:18.751504 13456 webserver.cc:460] Webserver started at http://127.31.235.131:44363/ using document root <none> and password file <none>
I20260620 20:25:18.752290 13456 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:18.752465 13456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 20:25:18 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987118, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
I20260620 20:25:18.778276 13456 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:25:18.781895 13473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:18.783631 13456 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 20:25:18.783806 13456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "ea7eb53221df4d3db69fdd03940e9f3f"
format_stamp: "Formatted at 2026-06-20 20:25:15 on dist-test-slave-zzwf"
server_key: "c9aaf74165bc4f1345b451f1eb13479d"
server_key_iv: "778eb4d414892d266199b5dd7fe57f75"
server_key_version: "kuduclusterkey@0"
I20260620 20:25:18.785265 13456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:18.811848 13456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:18.817065 13456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:18.817437 13456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:18.821274 13456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:25:18.825825 13456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:25:18.825946 13456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:18.826124 13456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:25:18.826212 13456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 20:25:18 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987118, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for HTTP/127.31.235.148@KRBTEST.COM
I20260620 20:25:18.952023 13456 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:38909
I20260620 20:25:18.952078 13586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:38909 every 8 connection(s)
I20260620 20:25:18.953994 13456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:25:18.960692 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 13456
I20260620 20:25:18.990916 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:18.961819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41710 (local address 127.31.235.190:42485)
0620 20:25:18.969997 (+ 8178us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:18.970008 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:18.970967 (+ 959us) server_negotiation.cc:407] Connection header received
0620 20:25:18.971074 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:18.971083 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:18.971205 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:18.971332 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:18.974057 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:18.975601 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:18.978323 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:18.979013 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:18.979138 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:18.982994 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:18.983001 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:18.983006 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:18.983076 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:18.986788 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:18.987699 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:18.987705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:18.987709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:18.987851 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:18.988257 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:18.988262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:18.988265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:18.988621 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:18.988884 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:18.990127 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0620 20:25:18.990346 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7976,"thread_start_us":7835,"threads_started":1}
Jun 20 20:25:19 dist-test-slave-zzwf krb5kdc[12094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987118, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:25:19.019688 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:18.978190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:42433 (local address 127.31.235.190:42485)
0620 20:25:18.991392 (+ 13202us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:18.991398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:18.996704 (+ 5306us) server_negotiation.cc:407] Connection header received
0620 20:25:19.000844 (+ 4140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:19.000851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:19.000950 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:19.001028 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:19.003344 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.004399 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.006813 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.007411 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.007530 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:19.012059 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:19.012066 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:19.012071 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:19.012112 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:19.015723 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:19.016742 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:19.016750 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:19.016756 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:19.016914 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:19.017460 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:19.017468 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:19.017473 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:19.017860 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:19.018006 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:19.019058 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0620 20:25:19.019314 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13093,"thread_start_us":20405,"threads_started":1}
I20260620 20:25:19.022019 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:18.977619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:42485 (local address 127.31.235.131:42433)
0620 20:25:18.996531 (+ 18912us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:18.996588 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:19.000460 (+ 3872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:19.001640 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:19.001654 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:19.002293 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:19.003106 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:19.003156 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.004627 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.004633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:19.006652 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:19.006667 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.007880 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.007889 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:19.008179 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:19.008994 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:19.009024 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:19.011777 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:19.015913 (+ 4136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:19.015923 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:19.015944 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:19.016594 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:19.017061 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:19.017066 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:19.017069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:19.017333 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:19.018103 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:19.018111 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:19.018603 (+ 492us) client_negotiation.cc:769] Sending connection context
0620 20:25:19.019969 (+ 1366us) client_negotiation.cc:240] Negotiation successful
0620 20:25:19.020575 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18573,"thread_start_us":18263,"threads_started":1}
I20260620 20:25:19.023553 13587 heartbeater.cc:347] Connected to a master server at 127.31.235.190:42485
I20260620 20:25:19.023869 13587 heartbeater.cc:464] Registering TS with master...
I20260620 20:25:19.024537 13587 heartbeater.cc:511] Master 127.31.235.190:42485 requested a full tablet report, sending...
I20260620 20:25:19.025942 13109 ts_manager.cc:194] Registered new tserver with Master: ea7eb53221df4d3db69fdd03940e9f3f (127.31.235.131:38909)
I20260620 20:25:19.027076 13109 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:42433
I20260620 20:25:19.031042 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:25:19.052901 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.038991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.31.235.190:42485)
0620 20:25:19.039195 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:19.039211 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:19.039735 (+ 524us) server_negotiation.cc:407] Connection header received
0620 20:25:19.039949 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:19.039955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:19.040033 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:19.040144 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:19.041555 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.042306 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.043774 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.044195 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.044324 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:19.047059 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:19.047065 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:19.047070 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:19.047110 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:19.049937 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:19.050577 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:19.050583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:19.050587 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:19.050707 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:19.051114 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:19.051119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:19.051122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:19.051475 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:19.051699 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:19.052212 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 20:25:19.052433 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:25:19.057777 13109 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:25:19.065953 13109 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:25:19.108006 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.094194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56378 (local address 127.31.235.129:35491)
0620 20:25:19.096210 (+ 2016us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:19.096222 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:19.096257 (+ 35us) server_negotiation.cc:407] Connection header received
0620 20:25:19.096456 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:19.096469 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:19.096771 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:19.097000 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:19.098123 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.100043 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.103761 (+ 3718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.105868 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.106075 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:19.106311 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:19.107243 (+ 932us) server_negotiation.cc:299] Negotiation successful
0620 20:25:19.107510 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1767,"thread_start_us":1558,"threads_started":1}
I20260620 20:25:19.109288 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.094799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39184 (local address 127.31.235.130:40263)
0620 20:25:19.097674 (+ 2875us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:19.097682 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:19.098247 (+ 565us) server_negotiation.cc:407] Connection header received
0620 20:25:19.098432 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:19.098442 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:19.098674 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:19.098883 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:19.100022 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.101932 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.105307 (+ 3375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.107395 (+ 2088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.107581 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:19.107782 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:19.108571 (+ 789us) server_negotiation.cc:299] Negotiation successful
0620 20:25:19.108806 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2745,"thread_start_us":2619,"threads_started":1}
I20260620 20:25:19.109810 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.093886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:35491 (local address 127.0.0.1:56378)
0620 20:25:19.094961 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:19.095001 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:19.095423 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:19.097111 (+ 1688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:19.097235 (+ 124us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:19.097286 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:19.097901 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:19.097923 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.100258 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.100266 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:19.103507 (+ 3241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:19.103531 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.106567 (+ 3036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.106585 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:19.106869 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:19.106938 (+ 69us) client_negotiation.cc:769] Sending connection context
0620 20:25:19.108652 (+ 1714us) client_negotiation.cc:240] Negotiation successful
0620 20:25:19.108956 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":3916,"threads_started":1}
I20260620 20:25:19.110261 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.094197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:40263 (local address 127.0.0.1:39184)
0620 20:25:19.098117 (+ 3920us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:19.098136 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:19.098285 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:19.099184 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:19.099192 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:19.099228 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:19.099775 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:19.099788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.102130 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.102134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:19.105081 (+ 2947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:19.105109 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.108034 (+ 2925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.108043 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:19.108316 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:19.108367 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 20:25:19.109538 (+ 1171us) client_negotiation.cc:240] Negotiation successful
0620 20:25:19.109737 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3747,"thread_start_us":750,"threads_started":1}
I20260620 20:25:19.114910 13387 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4aa2585be541068f990175e4955345 (DEFAULT_TABLE table=test-table [id=8baf5bfdd10c4b2cb4332740332745ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:25:19.114964 13242 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4aa2585be541068f990175e4955345 (DEFAULT_TABLE table=test-table [id=8baf5bfdd10c4b2cb4332740332745ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:25:19.116168 13387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4aa2585be541068f990175e4955345. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:25:19.116333 13242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4aa2585be541068f990175e4955345. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:25:19.119076 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.098574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:38909 (local address 127.0.0.1:49850)
0620 20:25:19.111113 (+ 12539us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:19.111133 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:19.111246 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:19.112068 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:19.112087 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:19.112119 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:19.112304 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:19.112314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.114596 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.114602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:19.116442 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:19.116457 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.118270 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.118276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:19.118468 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:19.118503 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 20:25:19.118623 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 20:25:19.118768 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12406,"thread_start_us":12950,"threads_started":1}
I20260620 20:25:19.119379 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.098844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49850 (local address 127.31.235.131:38909)
0620 20:25:19.111179 (+ 12335us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:19.111187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:19.111225 (+ 38us) server_negotiation.cc:407] Connection header received
0620 20:25:19.111542 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:19.111552 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:19.111817 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:19.112475 (+ 658us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:19.112767 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.114423 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.116620 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.118088 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.118297 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:19.118462 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:19.118668 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 20:25:19.118835 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12143,"thread_start_us":12400,"threads_started":1}
I20260620 20:25:19.123090 13522 tablet_service.cc:1467] Processing CreateTablet for tablet 2f4aa2585be541068f990175e4955345 (DEFAULT_TABLE table=test-table [id=8baf5bfdd10c4b2cb4332740332745ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:25:19.124555 13522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f4aa2585be541068f990175e4955345. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:25:19.129565 13616 tablet_bootstrap.cc:492] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef: Bootstrap starting.
I20260620 20:25:19.133766 13616 tablet_bootstrap.cc:654] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef: Neither blocks nor log segments found. Creating new log.
I20260620 20:25:19.134186 13617 tablet_bootstrap.cc:492] T 2f4aa2585be541068f990175e4955345 P f7a27a55420c4407a6613a3b063e1e1a: Bootstrap starting.
I20260620 20:25:19.135257 13616 log.cc:824] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef: Log is configured to *not* fsync() on all Append() calls
I20260620 20:25:19.137681 13617 tablet_bootstrap.cc:654] T 2f4aa2585be541068f990175e4955345 P f7a27a55420c4407a6613a3b063e1e1a: Neither blocks nor log segments found. Creating new log.
I20260620 20:25:19.138598 13616 tablet_bootstrap.cc:492] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef: No bootstrap required, opened a new log
I20260620 20:25:19.138876 13616 ts_tablet_manager.cc:1397] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 20:25:19.138888 13617 log.cc:824] T 2f4aa2585be541068f990175e4955345 P f7a27a55420c4407a6613a3b063e1e1a: Log is configured to *not* fsync() on all Append() calls
I20260620 20:25:19.140750 13619 tablet_bootstrap.cc:492] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f: Bootstrap starting.
I20260620 20:25:19.144174 13617 tablet_bootstrap.cc:492] T 2f4aa2585be541068f990175e4955345 P f7a27a55420c4407a6613a3b063e1e1a: No bootstrap required, opened a new log
I20260620 20:25:19.144222 13619 tablet_bootstrap.cc:654] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f: Neither blocks nor log segments found. Creating new log.
I20260620 20:25:19.144467 13617 ts_tablet_manager.cc:1397] T 2f4aa2585be541068f990175e4955345 P f7a27a55420c4407a6613a3b063e1e1a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260620 20:25:19.145267 13619 log.cc:824] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:25:19.147866 13619 tablet_bootstrap.cc:492] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f: No bootstrap required, opened a new log
I20260620 20:25:19.148167 13619 ts_tablet_manager.cc:1397] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 20:25:19.154361 13616 raft_consensus.cc:348] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a27a55420c4407a6613a3b063e1e1a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35491 } } peers { permanent_uuid: "8b09f8b27d46413586e1387dc3c849ef" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40263 } } peers { permanent_uuid: "ea7eb53221df4d3db69fdd03940e9f3f" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 38909 } }
I20260620 20:25:19.154969 13616 raft_consensus.cc:374] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:25:19.154950 13617 raft_consensus.cc:348] T 2f4aa2585be541068f990175e4955345 P f7a27a55420c4407a6613a3b063e1e1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a27a55420c4407a6613a3b063e1e1a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35491 } } peers { permanent_uuid: "8b09f8b27d46413586e1387dc3c849ef" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40263 } } peers { permanent_uuid: "ea7eb53221df4d3db69fdd03940e9f3f" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 38909 } }
I20260620 20:25:19.155174 13616 raft_consensus.cc:729] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b09f8b27d46413586e1387dc3c849ef, State: Initialized, Role: FOLLOWER
I20260620 20:25:19.155311 13617 raft_consensus.cc:374] T 2f4aa2585be541068f990175e4955345 P f7a27a55420c4407a6613a3b063e1e1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:25:19.155397 13617 raft_consensus.cc:729] T 2f4aa2585be541068f990175e4955345 P f7a27a55420c4407a6613a3b063e1e1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7a27a55420c4407a6613a3b063e1e1a, State: Initialized, Role: FOLLOWER
I20260620 20:25:19.156054 13617 consensus_queue.cc:260] T 2f4aa2585be541068f990175e4955345 P f7a27a55420c4407a6613a3b063e1e1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a27a55420c4407a6613a3b063e1e1a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35491 } } peers { permanent_uuid: "8b09f8b27d46413586e1387dc3c849ef" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40263 } } peers { permanent_uuid: "ea7eb53221df4d3db69fdd03940e9f3f" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 38909 } }
I20260620 20:25:19.156018 13619 raft_consensus.cc:348] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a27a55420c4407a6613a3b063e1e1a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35491 } } peers { permanent_uuid: "8b09f8b27d46413586e1387dc3c849ef" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40263 } } peers { permanent_uuid: "ea7eb53221df4d3db69fdd03940e9f3f" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 38909 } }
I20260620 20:25:19.156392 13619 raft_consensus.cc:374] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:25:19.156490 13619 raft_consensus.cc:729] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea7eb53221df4d3db69fdd03940e9f3f, State: Initialized, Role: FOLLOWER
I20260620 20:25:19.156070 13616 consensus_queue.cc:260] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a27a55420c4407a6613a3b063e1e1a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35491 } } peers { permanent_uuid: "8b09f8b27d46413586e1387dc3c849ef" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40263 } } peers { permanent_uuid: "ea7eb53221df4d3db69fdd03940e9f3f" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 38909 } }
I20260620 20:25:19.157270 13619 consensus_queue.cc:260] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a27a55420c4407a6613a3b063e1e1a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35491 } } peers { permanent_uuid: "8b09f8b27d46413586e1387dc3c849ef" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40263 } } peers { permanent_uuid: "ea7eb53221df4d3db69fdd03940e9f3f" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 38909 } }
I20260620 20:25:19.164014 13617 ts_tablet_manager.cc:1428] T 2f4aa2585be541068f990175e4955345 P f7a27a55420c4407a6613a3b063e1e1a: Time spent starting tablet: real 0.019s user 0.017s sys 0.003s
I20260620 20:25:19.164343 13307 heartbeater.cc:503] Master 127.31.235.190:42485 was elected leader, sending a full tablet report...
I20260620 20:25:19.167053 13587 heartbeater.cc:503] Master 127.31.235.190:42485 was elected leader, sending a full tablet report...
I20260620 20:25:19.167436 13616 ts_tablet_manager.cc:1428] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef: Time spent starting tablet: real 0.028s user 0.015s sys 0.008s
I20260620 20:25:19.167671 13619 ts_tablet_manager.cc:1428] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f: Time spent starting tablet: real 0.019s user 0.016s sys 0.004s
I20260620 20:25:19.168184 13453 heartbeater.cc:503] Master 127.31.235.190:42485 was elected leader, sending a full tablet report...
W20260620 20:25:19.226850 13588 tablet.cc:2369] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:25:19.250918 13308 tablet.cc:2369] T 2f4aa2585be541068f990175e4955345 P f7a27a55420c4407a6613a3b063e1e1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:25:19.385928 13624 raft_consensus.cc:482] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:25:19.386276 13624 raft_consensus.cc:504] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a27a55420c4407a6613a3b063e1e1a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35491 } } peers { permanent_uuid: "8b09f8b27d46413586e1387dc3c849ef" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40263 } } peers { permanent_uuid: "ea7eb53221df4d3db69fdd03940e9f3f" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 38909 } }
I20260620 20:25:19.387953 13624 leader_election.cc:288] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7a27a55420c4407a6613a3b063e1e1a (127.31.235.129:35491), 8b09f8b27d46413586e1387dc3c849ef (127.31.235.130:40263)
I20260620 20:25:19.394934 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.388391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:49021 (local address 127.31.235.130:40263)
0620 20:25:19.388608 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:19.388615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:19.389119 (+ 504us) server_negotiation.cc:407] Connection header received
0620 20:25:19.389232 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:19.389241 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:19.389316 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:19.389417 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:19.390085 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.391121 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.392859 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.393970 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.394160 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:19.394246 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:19.394537 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 20:25:19.394636 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:25:19.395004 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.388444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:40263 (local address 127.31.235.131:49021)
0620 20:25:19.389015 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:19.389029 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:19.389130 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:19.389444 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:19.389449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:19.389472 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:19.389896 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:19.389905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.391277 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.391281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:19.392731 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:19.392745 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.394220 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.394232 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:19.394389 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:19.394441 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 20:25:19.394513 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 20:25:19.394610 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":294,"threads_started":1}
I20260620 20:25:19.395138 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.388341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:60475 (local address 127.31.235.129:35491)
0620 20:25:19.388573 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:19.388579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:19.388821 (+ 242us) server_negotiation.cc:407] Connection header received
0620 20:25:19.389061 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:19.389065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:19.389165 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:19.389260 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:19.389724 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.390893 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.392735 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.393815 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.393996 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:19.394075 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:19.394748 (+ 673us) server_negotiation.cc:299] Negotiation successful
0620 20:25:19.394879 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:25:19.394966 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.388400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:35491 (local address 127.31.235.131:60475)
0620 20:25:19.388719 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:19.388735 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:19.388901 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:19.389272 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:19.389275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:19.389291 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:19.389558 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:19.389577 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.391114 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.391122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:19.392562 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:19.392575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.393997 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.394003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:19.394318 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:19.394352 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 20:25:19.394444 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 20:25:19.394587 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":1}
I20260620 20:25:19.396185 13408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4aa2585be541068f990175e4955345" candidate_uuid: "ea7eb53221df4d3db69fdd03940e9f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b09f8b27d46413586e1387dc3c849ef" is_pre_election: true
I20260620 20:25:19.396514 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4aa2585be541068f990175e4955345" candidate_uuid: "ea7eb53221df4d3db69fdd03940e9f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7a27a55420c4407a6613a3b063e1e1a" is_pre_election: true
I20260620 20:25:19.396723 13408 raft_consensus.cc:2454] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea7eb53221df4d3db69fdd03940e9f3f in term 0.
I20260620 20:25:19.397066 13262 raft_consensus.cc:2454] T 2f4aa2585be541068f990175e4955345 P f7a27a55420c4407a6613a3b063e1e1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea7eb53221df4d3db69fdd03940e9f3f in term 0.
I20260620 20:25:19.397533 13477 leader_election.cc:302] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b09f8b27d46413586e1387dc3c849ef, ea7eb53221df4d3db69fdd03940e9f3f; no voters:
I20260620 20:25:19.398010 13624 raft_consensus.cc:2781] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:25:19.398167 13624 raft_consensus.cc:482] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:25:19.398277 13624 raft_consensus.cc:3037] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:25:19.400514 13624 raft_consensus.cc:504] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a27a55420c4407a6613a3b063e1e1a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35491 } } peers { permanent_uuid: "8b09f8b27d46413586e1387dc3c849ef" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40263 } } peers { permanent_uuid: "ea7eb53221df4d3db69fdd03940e9f3f" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 38909 } }
I20260620 20:25:19.401051 13624 leader_election.cc:288] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f [CANDIDATE]: Term 1 election: Requested vote from peers f7a27a55420c4407a6613a3b063e1e1a (127.31.235.129:35491), 8b09f8b27d46413586e1387dc3c849ef (127.31.235.130:40263)
I20260620 20:25:19.401571 13408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4aa2585be541068f990175e4955345" candidate_uuid: "ea7eb53221df4d3db69fdd03940e9f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b09f8b27d46413586e1387dc3c849ef"
I20260620 20:25:19.401566 13262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f4aa2585be541068f990175e4955345" candidate_uuid: "ea7eb53221df4d3db69fdd03940e9f3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7a27a55420c4407a6613a3b063e1e1a"
I20260620 20:25:19.401778 13262 raft_consensus.cc:3037] T 2f4aa2585be541068f990175e4955345 P f7a27a55420c4407a6613a3b063e1e1a [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:25:19.401778 13408 raft_consensus.cc:3037] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef [term 0 FOLLOWER]: Advancing to term 1
W20260620 20:25:19.403973 13454 tablet.cc:2369] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:25:19.404901 13408 raft_consensus.cc:2454] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea7eb53221df4d3db69fdd03940e9f3f in term 1.
I20260620 20:25:19.405336 13477 leader_election.cc:302] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b09f8b27d46413586e1387dc3c849ef, ea7eb53221df4d3db69fdd03940e9f3f; no voters:
I20260620 20:25:19.405652 13624 raft_consensus.cc:2781] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:25:19.406170 13624 raft_consensus.cc:686] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f [term 1 LEADER]: Becoming Leader. State: Replica: ea7eb53221df4d3db69fdd03940e9f3f, State: Running, Role: LEADER
I20260620 20:25:19.406646 13624 consensus_queue.cc:237] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a27a55420c4407a6613a3b063e1e1a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35491 } } peers { permanent_uuid: "8b09f8b27d46413586e1387dc3c849ef" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40263 } } peers { permanent_uuid: "ea7eb53221df4d3db69fdd03940e9f3f" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 38909 } }
I20260620 20:25:19.406806 13262 raft_consensus.cc:2454] T 2f4aa2585be541068f990175e4955345 P f7a27a55420c4407a6613a3b063e1e1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea7eb53221df4d3db69fdd03940e9f3f in term 1.
I20260620 20:25:19.411674 13108 catalog_manager.cc:5495] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f reported cstate change: term changed from 0 to 1, leader changed from <none> to ea7eb53221df4d3db69fdd03940e9f3f (127.31.235.131). New cstate: current_term: 1 leader_uuid: "ea7eb53221df4d3db69fdd03940e9f3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7a27a55420c4407a6613a3b063e1e1a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b09f8b27d46413586e1387dc3c849ef" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea7eb53221df4d3db69fdd03940e9f3f" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 38909 } health_report { overall_health: HEALTHY } } }
I20260620 20:25:19.487947 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.482265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49866 (local address 127.31.235.131:38909)
0620 20:25:19.482456 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:19.482462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:19.482510 (+ 48us) server_negotiation.cc:407] Connection header received
0620 20:25:19.482774 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:19.482778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:19.482867 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:19.483030 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:25:19.483399 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.484165 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.485777 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.486193 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.486352 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:19.486894 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:25:19.487434 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:25:19.487547 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:19.487687 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 20:25:19.487752 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 20:25:19.496973 13408 raft_consensus.cc:1261] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef [term 1 FOLLOWER]: Refusing update from remote peer ea7eb53221df4d3db69fdd03940e9f3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:25:19.497309 13262 raft_consensus.cc:1261] T 2f4aa2585be541068f990175e4955345 P f7a27a55420c4407a6613a3b063e1e1a [term 1 FOLLOWER]: Refusing update from remote peer ea7eb53221df4d3db69fdd03940e9f3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:25:19.497961 13624 consensus_queue.cc:1035] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b09f8b27d46413586e1387dc3c849ef" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:25:19.498235 13626 consensus_queue.cc:1035] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7a27a55420c4407a6613a3b063e1e1a" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 35491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:25:19.510110 13629 mvcc.cc:205] Tried to move back new op lower bound from 7299019241448726528 to 7299019241093976064. Current Snapshot: MvccSnapshot[applied={T|T < 7299019241448726528}]
I20260620 20:25:19.511664 13631 mvcc.cc:205] Tried to move back new op lower bound from 7299019241448726528 to 7299019241093976064. Current Snapshot: MvccSnapshot[applied={T|T < 7299019241448726528}]
I20260620 20:25:19.529249 13108 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:25:19.529598 13108 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:25:19.533787 13108 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0058323f253a445daf2ea4d1a05fd9e1: Sending DeleteTablet for 3 replicas of tablet 2f4aa2585be541068f990175e4955345
I20260620 20:25:19.535034 13242 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f4aa2585be541068f990175e4955345 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:25:19 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:56378
I20260620 20:25:19.535087 13387 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f4aa2585be541068f990175e4955345 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:25:19 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:39184
I20260620 20:25:19.535778 13522 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f4aa2585be541068f990175e4955345 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:25:19 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:49850
I20260620 20:25:19.537412 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 13146
I20260620 20:25:19.554080 13640 tablet_replica.cc:330] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef: stopping tablet replica
I20260620 20:25:19.554720 13640 raft_consensus.cc:2229] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:25:19.555402 13640 raft_consensus.cc:2258] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:25:19.556447 13641 tablet_replica.cc:330] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f: stopping tablet replica
I20260620 20:25:19.556983 13641 raft_consensus.cc:2229] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:25:19.557919 13641 raft_consensus.cc:2258] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:25:19.561988 13641 ts_tablet_manager.cc:1905] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:25:19.564183 13640 ts_tablet_manager.cc:1905] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:25:19.565541 13639 tablet_replica.cc:330] T 2f4aa2585be541068f990175e4955345 P f7a27a55420c4407a6613a3b063e1e1a: stopping tablet replica
I20260620 20:25:19.565964 13639 raft_consensus.cc:2229] T 2f4aa2585be541068f990175e4955345 P f7a27a55420c4407a6613a3b063e1e1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:25:19.566444 13639 raft_consensus.cc:2258] T 2f4aa2585be541068f990175e4955345 P f7a27a55420c4407a6613a3b063e1e1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:25:19.571233 13641 ts_tablet_manager.cc:1918] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:25:19.571403 13640 ts_tablet_manager.cc:1918] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:25:19.571655 13640 log.cc:1196] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/2f4aa2585be541068f990175e4955345
I20260620 20:25:19.571774 13641 log.cc:1196] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/2f4aa2585be541068f990175e4955345
I20260620 20:25:19.572297 13640 ts_tablet_manager.cc:1939] T 2f4aa2585be541068f990175e4955345 P 8b09f8b27d46413586e1387dc3c849ef: Deleting consensus metadata
I20260620 20:25:19.572831 13641 ts_tablet_manager.cc:1939] T 2f4aa2585be541068f990175e4955345 P ea7eb53221df4d3db69fdd03940e9f3f: Deleting consensus metadata
I20260620 20:25:19.574426 13096 catalog_manager.cc:4841] TS 8b09f8b27d46413586e1387dc3c849ef (127.31.235.130:40263): tablet 2f4aa2585be541068f990175e4955345 (table test-table [id=8baf5bfdd10c4b2cb4332740332745ef]) successfully deleted
I20260620 20:25:19.574510 13095 catalog_manager.cc:4841] TS ea7eb53221df4d3db69fdd03940e9f3f (127.31.235.131:38909): tablet 2f4aa2585be541068f990175e4955345 (table test-table [id=8baf5bfdd10c4b2cb4332740332745ef]) successfully deleted
I20260620 20:25:19.575418 13639 ts_tablet_manager.cc:1905] T 2f4aa2585be541068f990175e4955345 P f7a27a55420c4407a6613a3b063e1e1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:25:19.577594 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.540966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56384 (local address 127.31.235.129:35491)
0620 20:25:19.541213 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:19.541222 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:19.550358 (+ 9136us) server_negotiation.cc:407] Connection header received
0620 20:25:19.550923 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:19.550933 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:19.551057 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:19.551148 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:19.558177 (+ 7029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.559412 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.561956 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.562636 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.562748 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:19.569291 (+ 6543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:19.569306 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:19.569324 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:19.569367 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:19.573407 (+ 4040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:19.574980 (+ 1573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:19.574990 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:19.574993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:19.575177 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:19.575687 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:19.575692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:19.575696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:19.576174 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:19.576505 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:19.576857 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 20:25:19.577134 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 20:25:19.578415 13302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:19.673003 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 13146
I20260620 20:25:19.690706 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 13311
W20260620 20:25:19.690850 13095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.235.129:35491 (error 108)
I20260620 20:25:19.697593 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.696638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:35491 (local address 127.0.0.1:56394)
0620 20:25:19.697194 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:19.697324 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":298,"threads_started":1}
W20260620 20:25:19.697894 13095 catalog_manager.cc:4584] TS f7a27a55420c4407a6613a3b063e1e1a (127.31.235.129:35491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f4aa2585be541068f990175e4955345: Network error: Client connection negotiation failed: client connection to 127.31.235.129:35491: connect: Connection refused (error 111)
I20260620 20:25:19.707042 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.691799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39194 (local address 127.31.235.130:40263)
0620 20:25:19.691980 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:19.691985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:19.692004 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:25:19.692065 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:19.692070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:19.692143 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:19.692246 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:19.694547 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.695681 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.697459 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.698063 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.698195 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:19.701017 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:19.701041 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:19.701057 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:19.701099 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:19.704041 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:19.704854 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:19.704864 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:19.704868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:19.705018 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:19.705393 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:19.705400 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:19.705403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:19.705841 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:19.706132 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:19.706459 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 20:25:19.706678 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:25:19.707700 13448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:19.751740 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.751142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:35491 (local address 127.0.0.1:56406)
0620 20:25:19.751441 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:19.751558 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260620 20:25:19.753113 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.752625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:35491 (local address 127.0.0.1:56416)
0620 20:25:19.752847 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:19.752930 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 20:25:19.814149 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.813622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:35491 (local address 127.0.0.1:56426)
0620 20:25:19.813852 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:19.813960 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 20:25:19.814177 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 13311
I20260620 20:25:19.816342 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.815848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:35491 (local address 127.0.0.1:56440)
0620 20:25:19.816045 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:19.816138 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 20:25:19.836404 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 13456
I20260620 20:25:19.851465 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.837321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.31.235.131:38909)
0620 20:25:19.837573 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:19.837580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:19.837599 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:25:19.837779 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:19.837784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:19.837864 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:19.838010 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:19.839334 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.840135 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.841431 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:19.841952 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:19.842095 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:19.844939 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:19.844953 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:19.844969 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:19.845013 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:19.848202 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:19.848994 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:19.849012 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:19.849015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:19.849174 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:19.849783 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:19.849790 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:19.849793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:19.850169 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:19.850528 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:19.850866 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 20:25:19.851093 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 20:25:19.852164 13582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:19.897778 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.897179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:35491 (local address 127.0.0.1:56446)
0620 20:25:19.897485 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:19.897626 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260620 20:25:19.899065 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:19.898606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:35491 (local address 127.0.0.1:56456)
0620 20:25:19.898785 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:19.898899 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 20:25:19.976763 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 13456
I20260620 20:25:19.996999 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 13076
I20260620 20:25:19.998543 13139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:20.113863 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 13076
I20260620 20:25:20.136104 32686 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 20:25:20.471697 32686 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 20:25:20.471951 32686 mini_ranger.cc:66] Stopping Ranger...
I20260620 20:25:20.818014 32686 mini_ranger.cc:68] Stopped Ranger
2026-06-20 20:25:20.818 UTC [12124] LOG: received smart shutdown request
2026-06-20 20:25:20.821 UTC [12124] LOG: background worker "logical replication launcher" (PID 12133) exited with exit code 1
2026-06-20 20:25:20.821 UTC [12128] LOG: shutting down
2026-06-20 20:25:21.010 UTC [12124] LOG: database system is shut down
2026-06-20T20:25:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53590 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:25:21 dist-test-slave-zzwf krb5kdc[13657](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:25:21 dist-test-slave-zzwf krb5kdc[13657](info): set up 2 sockets
Jun 20 20:25:21 dist-test-slave-zzwf krb5kdc[13657](info): commencing operation
krb5kdc: starting...
W20260620 20:25:22.302284 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:25:22 dist-test-slave-zzwf krb5kdc[13657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:25:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:25:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:25:22.542369 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:33343
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:35819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:33343,127.31.235.189:36085,127.31.235.188:45817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:25:22.880144 13670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:22.880450 13670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:22.880643 13670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:22.888168 13670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:25:22.888290 13670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:22.888398 13670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:22.888505 13670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:25:22.888592 13670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:25:22.898989 13670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:35819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:33343,127.31.235.189:36085,127.31.235.188:45817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:33343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:22.900362 13670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:22.901755 13670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:22.914572 13675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:22.914628 13678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:22.914587 13676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:22.915791 13670 server_base.cc:1034] running on GCE node
I20260620 20:25:22.916675 13670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:22.918258 13670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:22.919500 13670 hybrid_clock.cc:648] HybridClock initialized: now 1781987122919448 us; error 49 us; skew 500 ppm
I20260620 20:25:22.919873 13670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:22 dist-test-slave-zzwf krb5kdc[13657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987122, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:22.929649 13670 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:25:22.931326 13670 webserver.cc:460] Webserver started at http://127.31.235.190:46027/ using document root <none> and password file <none>
I20260620 20:25:22.932036 13670 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:22.932142 13670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:22.932401 13670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:25:22.935580 13670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "de140a4b29724e76ab6bfad839e43409"
format_stamp: "Formatted at 2026-06-20 20:25:22 on dist-test-slave-zzwf"
server_key: "8b0c02fab79e4fc5869e54c33ebd2887"
server_key_iv: "de6c54e0c71ca6996f8291a1717f9702"
server_key_version: "encryptionkey@0"
I20260620 20:25:22.936368 13670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "de140a4b29724e76ab6bfad839e43409"
format_stamp: "Formatted at 2026-06-20 20:25:22 on dist-test-slave-zzwf"
server_key: "8b0c02fab79e4fc5869e54c33ebd2887"
server_key_iv: "de6c54e0c71ca6996f8291a1717f9702"
server_key_version: "encryptionkey@0"
I20260620 20:25:22.941802 13670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 20:25:22.945246 13685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:22.946745 13670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:25:22.946934 13670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "de140a4b29724e76ab6bfad839e43409"
format_stamp: "Formatted at 2026-06-20 20:25:22 on dist-test-slave-zzwf"
server_key: "8b0c02fab79e4fc5869e54c33ebd2887"
server_key_iv: "de6c54e0c71ca6996f8291a1717f9702"
server_key_version: "encryptionkey@0"
I20260620 20:25:22.947154 13670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:22.966172 13670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:22.970327 13670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:22.970630 13670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:22.990154 13670 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:33343
I20260620 20:25:22.990187 13737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:33343 every 8 connection(s)
I20260620 20:25:22.991523 13670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:25:22.995286 13738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:25:22.995945 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 13670
I20260620 20:25:22.996170 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:25:22.997023 32686 external_mini_cluster.cc:1428] Setting key a12628d09db465efacb47ee9149702ad
I20260620 20:25:23.005852 13738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.235.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.189@KRBTEST.COM" created.
Jun 20 20:25:23 dist-test-slave-zzwf krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987122, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:25:23.031334 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.009224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33343 (local address 127.0.0.1:54966)
0620 20:25:23.010103 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.010144 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:23.012331 (+ 2187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:23.013188 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:23.013195 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:23.013942 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:23.014275 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:23.014291 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:23.015952 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:23.015957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:23.017048 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:23.017060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:23.018081 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:23.018087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:23.018290 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:23.018971 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:23.018992 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:23.023216 (+ 4224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:23.026971 (+ 3755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:23.026987 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:23.027002 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:23.027606 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:23.028177 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:23.028196 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:23.028200 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:23.028389 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:23.029438 (+ 1049us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:23.029444 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:23.029696 (+ 252us) client_negotiation.cc:769] Sending connection context
0620 20:25:23.029895 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 20:25:23.030275 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":399,"threads_started":1}
I20260620 20:25:23.031363 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.010666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54966 (local address 127.31.235.190:33343)
0620 20:25:23.011416 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:23.011428 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:23.011486 (+ 58us) server_negotiation.cc:407] Connection header received
0620 20:25:23.012719 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:23.012725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:23.012963 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:23.013178 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:23.014478 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:23.015803 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:23.017193 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:23.017836 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:23.018042 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:23.023477 (+ 5435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:23.023487 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:23.023501 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:23.023534 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:23.026820 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:23.027771 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:23.027777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:23.027780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:23.027961 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:23.028518 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:23.028527 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:23.028531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:23.029294 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:23.029568 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:23.030084 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 20:25:23.030458 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":543,"thread_start_us":332,"threads_started":1}
I20260620 20:25:23.034638 13738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:25:23.035841 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.035440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:36085 (local address 127.0.0.1:47186)
0620 20:25:23.035608 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.035722 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:36085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 20:25:23.036248 13689 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.235.189:36085: connect: Connection refused (error 111)
I20260620 20:25:23.037811 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.037455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:36085 (local address 127.0.0.1:47202)
0620 20:25:23.037610 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.037692 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:36085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 20:25:23.038045 13738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.189:36085: Network error: Client connection negotiation failed: client connection to 127.31.235.189:36085: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.189.keytab.
Entry for principal kudu/127.31.235.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.189.keytab.
I20260620 20:25:23.053000 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.189:36085
--webserver_interface=127.31.235.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.189.keytab
--principal=kudu/127.31.235.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:35819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:33343,127.31.235.189:36085,127.31.235.188:45817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:25:23.087451 13738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } attempt: 1
I20260620 20:25:23.088827 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.088253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:36085 (local address 127.0.0.1:47204)
0620 20:25:23.088515 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.088650 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:36085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 20:25:23.090135 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.089705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:36085 (local address 127.0.0.1:47206)
0620 20:25:23.089880 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.089977 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:36085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 20:25:23.090366 13738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.189:36085: Network error: Client connection negotiation failed: client connection to 127.31.235.189:36085: connect: Connection refused (error 111)
I20260620 20:25:23.158746 13738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } attempt: 2
I20260620 20:25:23.160161 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.159492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:36085 (local address 127.0.0.1:47212)
0620 20:25:23.159812 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.159961 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:36085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 20:25:23.161415 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.160839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:36085 (local address 127.0.0.1:47226)
0620 20:25:23.161053 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.161186 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:36085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 20:25:23.161690 13738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.189:36085: Network error: Client connection negotiation failed: client connection to 127.31.235.189:36085: connect: Connection refused (error 111)
I20260620 20:25:23.253036 13738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } attempt: 3
I20260620 20:25:23.254416 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.253831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:36085 (local address 127.0.0.1:47232)
0620 20:25:23.254088 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.254227 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:36085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 20:25:23.255693 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.255258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:36085 (local address 127.0.0.1:47238)
0620 20:25:23.255415 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.255528 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:36085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 20:25:23.255936 13738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.189:36085: Network error: Client connection negotiation failed: client connection to 127.31.235.189:36085: connect: Connection refused (error 111)
W20260620 20:25:23.393592 13745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:23.393898 13745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:23.393989 13745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 20:25:23.399224 13738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } attempt: 4
I20260620 20:25:23.400568 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.399932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:36085 (local address 127.0.0.1:47252)
0620 20:25:23.400277 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.400390 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:36085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260620 20:25:23.401779 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.401331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:36085 (local address 127.0.0.1:47254)
0620 20:25:23.401524 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.401631 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:36085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 20:25:23.401882 13745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:25:23.401984 13745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:23.402076 13745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:23.402213 13745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:25:23.402164 13738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.189:36085: Network error: Client connection negotiation failed: client connection to 127.31.235.189:36085: connect: Connection refused (error 111)
W20260620 20:25:23.402299 13745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:25:23.412570 13745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:35819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:33343,127.31.235.189:36085,127.31.235.188:45817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.189.keytab
--principal=kudu/127.31.235.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.189:36085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.235.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:23.414048 13745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:23.415400 13745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:23.427965 13750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:23.427901 13753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:23.427942 13751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:23.428705 13745 server_base.cc:1034] running on GCE node
I20260620 20:25:23.429550 13745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:23.431041 13745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:23.432292 13745 hybrid_clock.cc:648] HybridClock initialized: now 1781987123432250 us; error 62 us; skew 500 ppm
I20260620 20:25:23.432680 13745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:23 dist-test-slave-zzwf krb5kdc[13657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987123, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:23.436059 13745 init.cc:377] Logged in from keytab as kudu/127.31.235.189@KRBTEST.COM (short username kudu)
I20260620 20:25:23.437670 13745 webserver.cc:460] Webserver started at http://127.31.235.189:45955/ using document root <none> and password file <none>
I20260620 20:25:23.438385 13745 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:23.438599 13745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:23.438983 13745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:25:23.442301 13745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-1/data/instance:
uuid: "6227782cf4294bc2813c2f1ea0f4229e"
format_stamp: "Formatted at 2026-06-20 20:25:23 on dist-test-slave-zzwf"
server_key: "3dffad5b667fc100b58aeba635685904"
server_key_iv: "ae06007a37c5b22dc7847d2d5ca3f822"
server_key_version: "encryptionkey@0"
I20260620 20:25:23.443153 13745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-1/wal/instance:
uuid: "6227782cf4294bc2813c2f1ea0f4229e"
format_stamp: "Formatted at 2026-06-20 20:25:23 on dist-test-slave-zzwf"
server_key: "3dffad5b667fc100b58aeba635685904"
server_key_iv: "ae06007a37c5b22dc7847d2d5ca3f822"
server_key_version: "encryptionkey@0"
I20260620 20:25:23.448925 13745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 20:25:23.452782 13760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:23.454331 13745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:25:23.454550 13745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-1/wal
uuid: "6227782cf4294bc2813c2f1ea0f4229e"
format_stamp: "Formatted at 2026-06-20 20:25:23 on dist-test-slave-zzwf"
server_key: "3dffad5b667fc100b58aeba635685904"
server_key_iv: "ae06007a37c5b22dc7847d2d5ca3f822"
server_key_version: "encryptionkey@0"
I20260620 20:25:23.454829 13745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:23.476042 13745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:23.480826 13745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:23.481076 13745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:23.499197 13745 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.189:36085
I20260620 20:25:23.499253 13812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.189:36085 every 8 connection(s)
I20260620 20:25:23.500577 13745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-1/data/info.pb
I20260620 20:25:23.504348 13813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:25:23.506105 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 13745
I20260620 20:25:23.506309 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-1/wal/instance
I20260620 20:25:23.506839 32686 external_mini_cluster.cc:1428] Setting key 17d587714c55eb2a9fa0c18c1f42732e
I20260620 20:25:23.511241 13813 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:25:23 dist-test-slave-zzwf krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987123, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.189@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.235.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.188@KRBTEST.COM" created.
I20260620 20:25:23.533705 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.514223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54968 (local address 127.31.235.190:33343)
0620 20:25:23.514407 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:23.514412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:23.515677 (+ 1265us) server_negotiation.cc:407] Connection header received
0620 20:25:23.517608 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:23.517614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:23.517710 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:23.517824 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:23.519315 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:23.519992 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:23.521559 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:23.522004 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:23.522160 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:23.526857 (+ 4697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:23.526862 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:23.526865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:23.526902 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:23.530268 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:23.531105 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:23.531115 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:23.531121 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:23.531280 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:23.531711 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:23.531719 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:23.531724 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:23.532175 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:23.532331 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:23.533105 (+ 774us) server_negotiation.cc:299] Negotiation successful
0620 20:25:23.533346 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:25:23.534778 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.514714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33343 (local address 127.0.0.1:54968)
0620 20:25:23.515503 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.515544 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:23.517306 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:23.518059 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:23.518068 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:23.518563 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:23.519124 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:23.519142 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:23.520158 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:23.520168 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:23.521388 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:23.521399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:23.522281 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:23.522291 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:23.522634 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:23.523430 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:23.523492 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:23.526594 (+ 3102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:23.530465 (+ 3871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:23.530473 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:23.530488 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:23.530959 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:23.531405 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:23.531409 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:23.531411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:23.531592 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:23.532307 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:23.532314 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:23.532688 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 20:25:23.532972 (+ 284us) client_negotiation.cc:240] Negotiation successful
0620 20:25:23.533566 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":345,"threads_started":1}
I20260620 20:25:23.536656 13813 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:25:23 dist-test-slave-zzwf krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987123, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.189@KRBTEST.COM for kudu/127.31.235.189@KRBTEST.COM
I20260620 20:25:23.553025 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.537599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:36085 (local address 127.0.0.1:47262)
0620 20:25:23.537805 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.537831 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:23.537939 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:23.538927 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:23.538935 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:23.539442 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:23.539675 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:23.539689 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:23.541104 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:23.541113 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:23.542746 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:23.542762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:23.543640 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:23.543648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:23.543908 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:23.544522 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:23.544549 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:23.547027 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:23.550349 (+ 3322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:23.550357 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:23.550361 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:23.550863 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:23.551245 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:23.551249 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:23.551253 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:23.551394 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:23.552048 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:23.552053 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:23.552195 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 20:25:23.552342 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 20:25:23.552627 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 20:25:23.553140 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.537880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.31.235.189:36085)
0620 20:25:23.538512 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:23.538520 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:23.538546 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:25:23.538586 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:23.538592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:23.538762 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:23.538872 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:23.539870 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:23.540919 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:23.542902 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:23.543478 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:23.543583 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:23.547232 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:23.547243 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:23.547258 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:23.547290 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:23.550230 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:23.550997 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:23.551004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:23.551006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:23.551125 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:23.551499 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:23.551510 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:23.551513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:23.551926 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:23.552148 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:23.552486 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 20:25:23.552738 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":495,"thread_start_us":368,"threads_started":1}
I20260620 20:25:23.555909 13813 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:25:23.557229 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.556771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:45817 (local address 127.0.0.1:49506)
0620 20:25:23.556955 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.557071 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 20:25:23.557585 13763 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
I20260620 20:25:23.559203 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.558928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:45817 (local address 127.0.0.1:49508)
0620 20:25:23.559032 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.559106 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.188.keytab.
Entry for principal kudu/127.31.235.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.188.keytab.
W20260620 20:25:23.559430 13813 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:45817: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
I20260620 20:25:23.560590 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.188:45817
--webserver_interface=127.31.235.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.188.keytab
--principal=kudu/127.31.235.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:35819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:33343,127.31.235.189:36085,127.31.235.188:45817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:25:23.608794 13813 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } attempt: 1
I20260620 20:25:23.610185 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.609683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:45817 (local address 127.0.0.1:49522)
0620 20:25:23.609916 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.610030 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 20:25:23.611284 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.610927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:45817 (local address 127.0.0.1:49538)
0620 20:25:23.611108 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.611176 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260620 20:25:23.611542 13813 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:45817: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
I20260620 20:25:23.679764 13813 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } attempt: 2
I20260620 20:25:23.681126 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.680581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:45817 (local address 127.0.0.1:49552)
0620 20:25:23.680829 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.680923 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260620 20:25:23.682242 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.681894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:45817 (local address 127.0.0.1:49558)
0620 20:25:23.682042 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.682126 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 20:25:23.682462 13813 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:45817: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
I20260620 20:25:23.701481 13738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } attempt: 5
Jun 20 20:25:23 dist-test-slave-zzwf krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987122, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for kudu/127.31.235.189@KRBTEST.COM
I20260620 20:25:23.715185 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.702651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.31.235.189:36085)
0620 20:25:23.702871 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:23.702875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:23.702890 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:25:23.702942 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:23.702946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:23.703023 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:23.703135 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:23.703939 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:23.704653 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:23.705912 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:23.706349 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:23.706498 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:23.709993 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:23.709998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:23.710002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:23.710035 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:23.712601 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:23.713271 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:23.713278 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:23.713281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:23.713403 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:23.713802 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:23.713809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:23.713812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:23.714089 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:23.714243 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:23.714637 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 20:25:23.714860 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 20:25:23.715209 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.702372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:36085 (local address 127.0.0.1:47264)
0620 20:25:23.702564 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.702587 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:23.702734 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:23.703173 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:23.703179 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:23.703583 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:23.703755 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:23.703764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:23.704798 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:23.704804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:23.705756 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:23.705769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:23.706524 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:23.706530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:23.706717 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:23.707450 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:23.707470 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:23.709780 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:23.712752 (+ 2972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:23.712758 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:23.712762 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:23.713151 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:23.713538 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:23.713543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:23.713547 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:23.713694 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:23.714206 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:23.714212 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:23.714371 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 20:25:23.714510 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 20:25:23.714828 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 20:25:23.716472 13738 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:25:23.717775 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.717264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:45817 (local address 127.0.0.1:49566)
0620 20:25:23.717530 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.717631 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260620 20:25:23.718859 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.718481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:45817 (local address 127.0.0.1:49572)
0620 20:25:23.718637 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.718738 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 20:25:23.719137 13738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:45817: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
I20260620 20:25:23.770406 13738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } attempt: 1
I20260620 20:25:23.771548 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.771062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:45817 (local address 127.0.0.1:49574)
0620 20:25:23.771281 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.771388 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 20:25:23.772538 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.772220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:45817 (local address 127.0.0.1:49590)
0620 20:25:23.772366 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.772443 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 20:25:23.772744 13738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:45817: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
I20260620 20:25:23.773661 13813 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } attempt: 3
I20260620 20:25:23.774526 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.774140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:45817 (local address 127.0.0.1:49592)
0620 20:25:23.774306 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.774395 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 20:25:23.775502 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.775117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:45817 (local address 127.0.0.1:49598)
0620 20:25:23.775335 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.775401 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260620 20:25:23.775714 13813 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:45817: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
I20260620 20:25:23.840945 13738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } attempt: 2
I20260620 20:25:23.842094 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.841622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:45817 (local address 127.0.0.1:49614)
0620 20:25:23.841826 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.841930 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 20:25:23.843042 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.842716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:45817 (local address 127.0.0.1:49622)
0620 20:25:23.842862 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.842949 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 20:25:23.843217 13738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:45817: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
W20260620 20:25:23.892807 13820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:23.893191 13820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:23.893354 13820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:23.900867 13820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:25:23.901015 13820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:23.901129 13820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:23.901233 13820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:25:23.901310 13820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:25:23.911540 13820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:35819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:33343,127.31.235.189:36085,127.31.235.188:45817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.188.keytab
--principal=kudu/127.31.235.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.188:45817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.235.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:23.912964 13820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:23.914327 13820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:25:23.918938 13813 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } attempt: 4
I20260620 20:25:23.920348 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.919636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:45817 (local address 127.0.0.1:49626)
0620 20:25:23.920005 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.920129 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260620 20:25:23.921605 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.921204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:45817 (local address 127.0.0.1:49632)
0620 20:25:23.921353 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.921425 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 20:25:23.921844 13813 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:45817: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
W20260620 20:25:23.926242 13828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:23.926316 13825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:23.926285 13826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:23.927443 13820 server_base.cc:1034] running on GCE node
I20260620 20:25:23.928112 13820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:23.929388 13820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:23.930646 13820 hybrid_clock.cc:648] HybridClock initialized: now 1781987123930612 us; error 71 us; skew 500 ppm
I20260620 20:25:23.931109 13820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:23 dist-test-slave-zzwf krb5kdc[13657](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987123, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:23.934283 13820 init.cc:377] Logged in from keytab as kudu/127.31.235.188@KRBTEST.COM (short username kudu)
I20260620 20:25:23.935735 13820 webserver.cc:460] Webserver started at http://127.31.235.188:42563/ using document root <none> and password file <none>
I20260620 20:25:23.936436 13820 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:23.936585 13820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:23.936901 13820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:25:23.939975 13820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-2/data/instance:
uuid: "f9e04238bf42421ba89556e550f867df"
format_stamp: "Formatted at 2026-06-20 20:25:23 on dist-test-slave-zzwf"
server_key: "28c425ba89847a3a637f5d5744f9e4c0"
server_key_iv: "d835b64566e3f9a5bab5c583d0d100ec"
server_key_version: "encryptionkey@0"
I20260620 20:25:23.940806 13820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-2/wal/instance:
uuid: "f9e04238bf42421ba89556e550f867df"
format_stamp: "Formatted at 2026-06-20 20:25:23 on dist-test-slave-zzwf"
server_key: "28c425ba89847a3a637f5d5744f9e4c0"
server_key_iv: "d835b64566e3f9a5bab5c583d0d100ec"
server_key_version: "encryptionkey@0"
I20260620 20:25:23.946141 13820 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:25:23.949445 13738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } attempt: 3
I20260620 20:25:23.949901 13835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:23.950632 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.950137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:45817 (local address 127.0.0.1:49646)
0620 20:25:23.950375 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.950459 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 20:25:23.951591 13820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:25:23.951800 13820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-2/wal
uuid: "f9e04238bf42421ba89556e550f867df"
format_stamp: "Formatted at 2026-06-20 20:25:23 on dist-test-slave-zzwf"
server_key: "28c425ba89847a3a637f5d5744f9e4c0"
server_key_iv: "d835b64566e3f9a5bab5c583d0d100ec"
server_key_version: "encryptionkey@0"
I20260620 20:25:23.952028 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:23.951455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:45817 (local address 127.0.0.1:49662)
0620 20:25:23.951689 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:23.951814 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260620 20:25:23.952044 13820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260620 20:25:23.952270 13738 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:45817: Network error: Client connection negotiation failed: client connection to 127.31.235.188:45817: connect: Connection refused (error 111)
I20260620 20:25:23.974107 13820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:23.978700 13820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:23.978948 13820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:23.997334 13820 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.188:45817
I20260620 20:25:23.997342 13886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.188:45817 every 8 connection(s)
I20260620 20:25:23.998646 13820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-2/data/info.pb
I20260620 20:25:24.002588 13887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:25:24.003453 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 13820
I20260620 20:25:24.003671 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781986916608810-32686-0/minicluster-data/master-2/wal/instance
I20260620 20:25:24.004166 32686 external_mini_cluster.cc:1428] Setting key 02ee0f90a3ae50104955777d6ed3ceea
I20260620 20:25:24.012955 13887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:25:24 dist-test-slave-zzwf krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.188@KRBTEST.COM
Jun 20 20:25:24 dist-test-slave-zzwf krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987123, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.188@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:25:24.030582 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:24.016069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54976 (local address 127.31.235.190:33343)
0620 20:25:24.016322 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:24.016330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:24.017015 (+ 685us) server_negotiation.cc:407] Connection header received
0620 20:25:24.017237 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:24.017241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:24.017330 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:24.017422 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:24.018730 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.019400 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.020683 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.021108 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.021248 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:24.024733 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:24.024738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:24.024741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:24.024774 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:24.027964 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.028607 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.028613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.028616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.028758 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.029099 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.029106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.029109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.029362 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:24.029548 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:24.029984 (+ 436us) server_negotiation.cc:299] Negotiation successful
0620 20:25:24.030193 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 20:25:24.031042 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:24.016036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:33343 (local address 127.0.0.1:54976)
0620 20:25:24.016862 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:24.016903 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:24.017068 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:24.017545 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:24.017558 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:24.018312 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:24.018559 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:24.018568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.019553 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.019558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:24.020532 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:24.020543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.021339 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.021348 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:24.021566 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:24.022251 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:24.022273 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:24.024527 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:24.028076 (+ 3549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:24.028087 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:24.028100 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:24.028486 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:24.028866 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:24.028872 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:24.028875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:24.028994 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:24.029514 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:24.029522 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:24.029690 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 20:25:24.029888 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 20:25:24.030186 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":409,"threads_started":1}
I20260620 20:25:24.031081 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:24.006061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49678 (local address 127.31.235.188:45817)
0620 20:25:24.006845 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:24.006861 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:24.006920 (+ 59us) server_negotiation.cc:407] Connection header received
0620 20:25:24.009567 (+ 2647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:24.009599 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:24.010198 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:24.010692 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:24.012504 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.014530 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.016684 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.017680 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.017887 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:24.022642 (+ 4755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:24.022651 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:24.022669 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:24.022701 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:24.026069 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.027031 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.027039 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.027048 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.027255 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.027773 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.027781 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.027787 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.028394 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:24.028719 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:24.029256 (+ 537us) server_negotiation.cc:299] Negotiation successful
0620 20:25:24.029859 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":322,"threads_started":1}
I20260620 20:25:24.033460 13887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:25:24 dist-test-slave-zzwf krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987123, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.188@KRBTEST.COM for kudu/127.31.235.189@KRBTEST.COM
Jun 20 20:25:24 dist-test-slave-zzwf krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:25:24.048070 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:24.034338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:36085 (local address 127.0.0.1:47278)
0620 20:25:24.034509 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:24.034527 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:24.034622 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:24.034974 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:24.034978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:24.035527 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:24.035760 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:24.035776 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.036919 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.036924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:24.038362 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:24.038378 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.039055 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.039063 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:24.039276 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:24.039908 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:24.039926 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:24.042344 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:24.045669 (+ 3325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:24.045676 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:24.045680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:24.046059 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:24.046448 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:24.046453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:24.046457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:24.046604 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:24.047112 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:24.047116 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:24.047265 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 20:25:24.047408 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 20:25:24.047699 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 20:25:24.048094 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:24.034469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47278 (local address 127.31.235.189:36085)
0620 20:25:24.034628 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:24.034634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:24.034653 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:25:24.034754 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:24.034760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:24.034829 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:24.034926 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:24.035941 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.036779 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.038502 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.038900 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.038993 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:24.042548 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:24.042554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:24.042557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:24.042588 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:24.045485 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.046180 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.046187 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.046191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.046329 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.046713 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.046720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.046723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.046992 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:24.047164 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:24.047507 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 20:25:24.047710 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 20:25:24.049285 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:24.034275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54986 (local address 127.31.235.190:33343)
0620 20:25:24.034665 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:24.034694 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:24.034712 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:25:24.034756 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:24.034760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:24.034829 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:24.034924 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:24.036480 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.037360 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.039141 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.039552 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.039701 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:24.043762 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:24.043768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:24.043773 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:24.043811 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:24.046292 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.046985 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.046990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.046994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.047149 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.047506 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.047512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.047515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.047774 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:24.048054 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:24.048720 (+ 666us) server_negotiation.cc:299] Negotiation successful
0620 20:25:24.048938 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260620 20:25:24.049242 13887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:25:24 dist-test-slave-zzwf krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987123, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.188@KRBTEST.COM for kudu/127.31.235.188@KRBTEST.COM
Jun 20 20:25:24 dist-test-slave-zzwf krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.189@KRBTEST.COM
I20260620 20:25:24.063405 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:24.050269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:45817 (local address 127.0.0.1:49690)
0620 20:25:24.050429 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:24.050443 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:24.050517 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:24.051226 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:24.051233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:24.051668 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:24.051849 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:24.051861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.052784 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.052791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:24.053767 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:24.053780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.054436 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.054442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:24.054648 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:24.055382 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:24.055413 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:24.057907 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:24.061043 (+ 3136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:24.061050 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:24.061054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:24.061360 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:24.061730 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:24.061736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:24.061739 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:24.061864 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:24.062375 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:24.062380 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:24.062524 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 20:25:24.062676 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 20:25:24.062963 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 20:25:24.063433 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:24.050329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49690 (local address 127.31.235.188:45817)
0620 20:25:24.050554 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:24.050559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:24.050576 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:25:24.050623 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:24.050628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:24.050696 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:24.050795 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:24.051994 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.052658 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.053907 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.054301 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.054390 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:24.058079 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:24.058085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:24.058089 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:24.058143 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:24.060925 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.061468 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.061476 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.061479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.061606 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.061962 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.061969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.061972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.062266 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:24.062412 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:24.062746 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 20:25:24.063100 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":146,"server-negotiator.queue_time_us":125}
I20260620 20:25:24.066744 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:24.051090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47286 (local address 127.31.235.189:36085)
0620 20:25:24.051256 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:24.051261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:24.051278 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:25:24.051412 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:24.051415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:24.051529 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:24.051603 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:24.052907 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.053662 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.055321 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.056052 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.056198 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:24.061320 (+ 5122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:24.061325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:24.061328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:24.061357 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:24.064144 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.064793 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.064800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.064803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.064974 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.065329 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.065335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.065338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.065656 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:24.065927 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:24.066211 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 20:25:24.066430 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 20:25:24.068431 13887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df: Bootstrap starting.
I20260620 20:25:24.071872 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:25:24.072300 13887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df: Neither blocks nor log segments found. Creating new log.
I20260620 20:25:24.073295 13887 log.cc:824] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df: Log is configured to *not* fsync() on all Append() calls
I20260620 20:25:24.076313 13887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df: No bootstrap required, opened a new log
I20260620 20:25:24.090854 13887 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de140a4b29724e76ab6bfad839e43409" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } } peers { permanent_uuid: "6227782cf4294bc2813c2f1ea0f4229e" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } } peers { permanent_uuid: "f9e04238bf42421ba89556e550f867df" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } }
I20260620 20:25:24.091518 13887 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:25:24.091696 13887 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e04238bf42421ba89556e550f867df, State: Initialized, Role: FOLLOWER
I20260620 20:25:24.092602 13887 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de140a4b29724e76ab6bfad839e43409" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } } peers { permanent_uuid: "6227782cf4294bc2813c2f1ea0f4229e" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } } peers { permanent_uuid: "f9e04238bf42421ba89556e550f867df" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } }
I20260620 20:25:24.099839 13900 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de140a4b29724e76ab6bfad839e43409" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } } peers { permanent_uuid: "6227782cf4294bc2813c2f1ea0f4229e" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } } peers { permanent_uuid: "f9e04238bf42421ba89556e550f867df" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } } }
I20260620 20:25:24.102566 13900 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:25:24.102434 13887 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:25:24.104072 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:24.077576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.31.235.190:33343)
0620 20:25:24.077757 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:24.077762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:24.084354 (+ 6592us) server_negotiation.cc:407] Connection header received
0620 20:25:24.084978 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:24.084989 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:24.085103 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:24.085199 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:24.088141 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.089277 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.091458 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.092141 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.092235 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:24.096476 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:24.096486 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:24.096494 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:24.096545 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:24.100265 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.101359 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.101364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.101368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.101587 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.102103 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.102107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.102110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.102479 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:24.102730 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:24.103457 (+ 727us) server_negotiation.cc:299] Negotiation successful
0620 20:25:24.103705 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:25:24.104089 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:24.077379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49692 (local address 127.31.235.188:45817)
0620 20:25:24.078573 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:24.078578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:24.086203 (+ 7625us) server_negotiation.cc:407] Connection header received
0620 20:25:24.086483 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:24.086489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:24.086576 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:24.086688 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:24.088766 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.089881 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.091980 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.092726 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.092836 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:24.096632 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:24.096638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:24.096644 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:24.096685 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:24.100495 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.101377 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.101383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.101386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.101562 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.102108 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.102113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.102115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.102532 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:24.102740 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:24.103457 (+ 717us) server_negotiation.cc:299] Negotiation successful
0620 20:25:24.103700 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1059}
I20260620 20:25:24.107462 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:24.076950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47290 (local address 127.31.235.189:36085)
0620 20:25:24.077129 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:24.077135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:24.086469 (+ 9334us) server_negotiation.cc:407] Connection header received
0620 20:25:24.086731 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:24.086737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:24.086814 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:24.086882 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:24.089160 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.090247 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.092406 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.093040 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.093145 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:24.099106 (+ 5961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:24.099116 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:24.099123 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:24.099178 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:24.102850 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.103799 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.103806 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.103809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.103975 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.104558 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.104567 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.104572 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.105085 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:24.106334 (+ 1249us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:24.107031 (+ 697us) server_negotiation.cc:299] Negotiation successful
0620 20:25:24.107218 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 20:25:24.119747 13911 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:25:24.119894 13911 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:25:24.116772 13820 master_runner.cc:186] Error getting master registration for 127.31.235.190:33343: OK, instance_id { permanent_uuid: "de140a4b29724e76ab6bfad839e43409" instance_seqno: 1781987122984602 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:25:24.128945 13820 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:25:24.129530 13738 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } attempt: 4
Jun 20 20:25:24 dist-test-slave-zzwf krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987122, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for kudu/127.31.235.188@KRBTEST.COM
I20260620 20:25:24.144788 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:24.130122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49700 (local address 127.31.235.188:45817)
0620 20:25:24.130289 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:24.130295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:24.130314 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:25:24.130425 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:24.130431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:24.130521 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:24.130628 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:24.131350 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.132618 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.133905 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.134584 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.134747 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:24.138916 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:24.138922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:24.138925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:24.138954 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:24.141775 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.142524 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.142528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.142530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.142699 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.143134 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.143140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.143143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.143532 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:24.143678 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:24.144191 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 20:25:24.144431 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:25:24.144783 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:24.130042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:45817 (local address 127.0.0.1:49700)
0620 20:25:24.130240 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:24.130257 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:24.130336 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:24.130652 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:24.130655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:24.131026 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:24.131207 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:24.131216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.132754 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.132758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:24.133775 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:24.133787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.134980 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.134989 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:24.135210 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:24.135995 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:24.136019 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:24.138726 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:24.141901 (+ 3175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:24.141906 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:24.141910 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:24.142406 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:24.142824 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:24.142829 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:24.142833 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:24.142994 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:24.143741 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:24.143747 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:24.143932 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 20:25:24.144097 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 20:25:24.144378 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 20:25:24.150791 13738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409: Bootstrap starting.
I20260620 20:25:24.154838 13738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409: Neither blocks nor log segments found. Creating new log.
I20260620 20:25:24.156301 13738 log.cc:824] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409: Log is configured to *not* fsync() on all Append() calls
I20260620 20:25:24.159536 13738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409: No bootstrap required, opened a new log
I20260620 20:25:24.170563 13738 raft_consensus.cc:348] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de140a4b29724e76ab6bfad839e43409" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } } peers { permanent_uuid: "6227782cf4294bc2813c2f1ea0f4229e" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } } peers { permanent_uuid: "f9e04238bf42421ba89556e550f867df" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } }
I20260620 20:25:24.170967 13738 raft_consensus.cc:374] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:25:24.171103 13738 raft_consensus.cc:729] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de140a4b29724e76ab6bfad839e43409, State: Initialized, Role: FOLLOWER
I20260620 20:25:24.171759 13738 consensus_queue.cc:260] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de140a4b29724e76ab6bfad839e43409" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } } peers { permanent_uuid: "6227782cf4294bc2813c2f1ea0f4229e" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } } peers { permanent_uuid: "f9e04238bf42421ba89556e550f867df" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } }
I20260620 20:25:24.172708 13914 sys_catalog.cc:455] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de140a4b29724e76ab6bfad839e43409" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } } peers { permanent_uuid: "6227782cf4294bc2813c2f1ea0f4229e" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } } peers { permanent_uuid: "f9e04238bf42421ba89556e550f867df" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } } }
I20260620 20:25:24.173053 13914 sys_catalog.cc:458] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:25:24.174091 13738 sys_catalog.cc:564] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 20:25:24.184449 13926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:25:24.184684 13926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:25:24.183102 13670 master_runner.cc:186] Error getting master registration for 127.31.235.189:36085: OK, instance_id { permanent_uuid: "6227782cf4294bc2813c2f1ea0f4229e" instance_seqno: 1781987123493968 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:25:24.193236 13670 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:25:24.221364 13813 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } attempt: 5
W20260620 20:25:24.223989 13691 tablet.cc:2369] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:25:24.231118 13841 tablet.cc:2369] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 20:25:24 dist-test-slave-zzwf krb5kdc[13657](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987123, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.189@KRBTEST.COM for kudu/127.31.235.188@KRBTEST.COM
I20260620 20:25:24.241645 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:24.222007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:45817 (local address 127.0.0.1:49702)
0620 20:25:24.222384 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:24.222406 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:24.222532 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:24.223470 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:24.223476 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:24.224049 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:24.224280 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:24.224293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.225860 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.225866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:24.227291 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:24.227306 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.228406 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.228412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:24.228638 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:24.229350 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:24.229383 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:24.232277 (+ 2894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:24.236919 (+ 4642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:24.236926 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:24.236931 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:24.237478 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:24.237925 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:24.237931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:24.237936 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:24.238146 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:24.238808 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:24.238814 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:24.238986 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 20:25:24.239161 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 20:25:24.241156 (+ 1995us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246}
I20260620 20:25:24.242393 13890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:24.222748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49702 (local address 127.31.235.188:45817)
0620 20:25:24.222931 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:24.222938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:24.222957 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:25:24.223022 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:24.223028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:24.223105 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:24.223215 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:24.224491 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.225687 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.227471 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:24.228235 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:24.228373 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:24.232504 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:24.232511 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:24.232517 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:24.232560 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:24.236760 (+ 4200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.237635 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.237641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.237645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.237795 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:24.238290 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:24.238295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:24.238299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:24.238682 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:24.238835 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:24.239273 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 20:25:24.239520 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:25:24.248443 13813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e: Bootstrap starting.
I20260620 20:25:24.252782 13813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e: Neither blocks nor log segments found. Creating new log.
I20260620 20:25:24.254170 13813 log.cc:824] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:25:24.258215 13813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e: No bootstrap required, opened a new log
I20260620 20:25:24.270889 13813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de140a4b29724e76ab6bfad839e43409" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } } peers { permanent_uuid: "6227782cf4294bc2813c2f1ea0f4229e" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } } peers { permanent_uuid: "f9e04238bf42421ba89556e550f867df" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } }
I20260620 20:25:24.271299 13813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:25:24.271451 13813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6227782cf4294bc2813c2f1ea0f4229e, State: Initialized, Role: FOLLOWER
I20260620 20:25:24.272106 13813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de140a4b29724e76ab6bfad839e43409" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } } peers { permanent_uuid: "6227782cf4294bc2813c2f1ea0f4229e" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } } peers { permanent_uuid: "f9e04238bf42421ba89556e550f867df" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } }
I20260620 20:25:24.273099 13929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de140a4b29724e76ab6bfad839e43409" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } } peers { permanent_uuid: "6227782cf4294bc2813c2f1ea0f4229e" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } } peers { permanent_uuid: "f9e04238bf42421ba89556e550f867df" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } } }
I20260620 20:25:24.273746 13929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:25:24.274480 13813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:25:24.303835 13745 master_runner.cc:263] No leader master found from master 6227782cf4294bc2813c2f1ea0f4229e
I20260620 20:25:24.303979 13745 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 20:25:24.308319 13940 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:25:24.308593 13940 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:25:24.405112 13900 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:25:24.405473 13900 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de140a4b29724e76ab6bfad839e43409" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } } peers { permanent_uuid: "6227782cf4294bc2813c2f1ea0f4229e" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } } peers { permanent_uuid: "f9e04238bf42421ba89556e550f867df" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } }
I20260620 20:25:24.407150 13900 leader_election.cc:288] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de140a4b29724e76ab6bfad839e43409 (127.31.235.190:33343), 6227782cf4294bc2813c2f1ea0f4229e (127.31.235.189:36085)
I20260620 20:25:24.408471 13713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9e04238bf42421ba89556e550f867df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de140a4b29724e76ab6bfad839e43409" is_pre_election: true
I20260620 20:25:24.409142 13713 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9e04238bf42421ba89556e550f867df in term 0.
I20260620 20:25:24.409936 13788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9e04238bf42421ba89556e550f867df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6227782cf4294bc2813c2f1ea0f4229e" is_pre_election: true
I20260620 20:25:24.410550 13788 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9e04238bf42421ba89556e550f867df in term 0.
I20260620 20:25:24.411284 13839 leader_election.cc:302] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6227782cf4294bc2813c2f1ea0f4229e, f9e04238bf42421ba89556e550f867df; no voters:
I20260620 20:25:24.412055 13900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:25:24.412225 13900 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:25:24.412339 13900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:25:24.417591 13900 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de140a4b29724e76ab6bfad839e43409" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } } peers { permanent_uuid: "6227782cf4294bc2813c2f1ea0f4229e" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } } peers { permanent_uuid: "f9e04238bf42421ba89556e550f867df" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } }
I20260620 20:25:24.419296 13713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9e04238bf42421ba89556e550f867df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de140a4b29724e76ab6bfad839e43409"
I20260620 20:25:24.419543 13713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:25:24.423439 13713 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e04238bf42421ba89556e550f867df in term 1.
I20260620 20:25:24.423995 13836 leader_election.cc:302] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de140a4b29724e76ab6bfad839e43409, f9e04238bf42421ba89556e550f867df; no voters:
I20260620 20:25:24.426987 13788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9e04238bf42421ba89556e550f867df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6227782cf4294bc2813c2f1ea0f4229e"
I20260620 20:25:24.427239 13788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:25:24.430972 13788 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e04238bf42421ba89556e550f867df in term 1.
I20260620 20:25:24.431845 13900 leader_election.cc:288] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [CANDIDATE]: Term 1 election: Requested vote from peers de140a4b29724e76ab6bfad839e43409 (127.31.235.190:33343), 6227782cf4294bc2813c2f1ea0f4229e (127.31.235.189:36085)
I20260620 20:25:24.432315 13900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:25:24.432782 13900 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [term 1 LEADER]: Becoming Leader. State: Replica: f9e04238bf42421ba89556e550f867df, State: Running, Role: LEADER
I20260620 20:25:24.434399 13900 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de140a4b29724e76ab6bfad839e43409" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } } peers { permanent_uuid: "6227782cf4294bc2813c2f1ea0f4229e" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } } peers { permanent_uuid: "f9e04238bf42421ba89556e550f867df" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } }
I20260620 20:25:24.455060 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9e04238bf42421ba89556e550f867df. Latest consensus state: current_term: 1 leader_uuid: "f9e04238bf42421ba89556e550f867df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de140a4b29724e76ab6bfad839e43409" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } } peers { permanent_uuid: "6227782cf4294bc2813c2f1ea0f4229e" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } } peers { permanent_uuid: "f9e04238bf42421ba89556e550f867df" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } } }
I20260620 20:25:24.455668 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [sys.catalog]: This master's current role is: LEADER
I20260620 20:25:24.462796 13944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:25:24.465209 13944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 20:25:24.483923 13766 tablet.cc:2369] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:25:24.485807 13788 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e [term 1 FOLLOWER]: Refusing update from remote peer f9e04238bf42421ba89556e550f867df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:25:24.486963 13942 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [LEADER]: Connected to new peer: Peer: permanent_uuid: "6227782cf4294bc2813c2f1ea0f4229e" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:25:24.495076 13713 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409 [term 1 FOLLOWER]: Refusing update from remote peer f9e04238bf42421ba89556e550f867df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:25:24.496241 13942 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [LEADER]: Connected to new peer: Peer: permanent_uuid: "de140a4b29724e76ab6bfad839e43409" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:25:24.551913 13914 sys_catalog.cc:455] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9e04238bf42421ba89556e550f867df. Latest consensus state: current_term: 1 leader_uuid: "f9e04238bf42421ba89556e550f867df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de140a4b29724e76ab6bfad839e43409" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } } peers { permanent_uuid: "6227782cf4294bc2813c2f1ea0f4229e" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } } peers { permanent_uuid: "f9e04238bf42421ba89556e550f867df" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } } }
I20260620 20:25:24.552296 13914 sys_catalog.cc:458] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:25:24.558844 13945 mvcc.cc:205] Tried to move back new op lower bound from 7299019261881393152 to 7299019261701382144. Current Snapshot: MvccSnapshot[applied={T|T < 7299019261881393152}]
I20260620 20:25:24.564657 13914 sys_catalog.cc:455] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f9e04238bf42421ba89556e550f867df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de140a4b29724e76ab6bfad839e43409" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } } peers { permanent_uuid: "6227782cf4294bc2813c2f1ea0f4229e" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } } peers { permanent_uuid: "f9e04238bf42421ba89556e550f867df" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } } }
I20260620 20:25:24.564916 13914 sys_catalog.cc:458] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:25:24.565351 13947 mvcc.cc:205] Tried to move back new op lower bound from 7299019261881393152 to 7299019261701382144. Current Snapshot: MvccSnapshot[applied={T|T < 7299019261881393152}]
I20260620 20:25:24.574663 13900 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f9e04238bf42421ba89556e550f867df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de140a4b29724e76ab6bfad839e43409" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } } peers { permanent_uuid: "6227782cf4294bc2813c2f1ea0f4229e" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } } peers { permanent_uuid: "f9e04238bf42421ba89556e550f867df" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } } }
I20260620 20:25:24.575024 13900 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [sys.catalog]: This master's current role is: LEADER
I20260620 20:25:24.580415 13944 catalog_manager.cc:1347] Generated new cluster ID: 92ec327c445b4790b618e8df7be9b8a3
I20260620 20:25:24.580533 13944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:25:24.595381 13929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9e04238bf42421ba89556e550f867df. Latest consensus state: current_term: 1 leader_uuid: "f9e04238bf42421ba89556e550f867df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de140a4b29724e76ab6bfad839e43409" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } } peers { permanent_uuid: "6227782cf4294bc2813c2f1ea0f4229e" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } } peers { permanent_uuid: "f9e04238bf42421ba89556e550f867df" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } } }
I20260620 20:25:24.595758 13929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:25:24.602434 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f9e04238bf42421ba89556e550f867df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de140a4b29724e76ab6bfad839e43409" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } } peers { permanent_uuid: "6227782cf4294bc2813c2f1ea0f4229e" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } } peers { permanent_uuid: "f9e04238bf42421ba89556e550f867df" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } } }
I20260620 20:25:24.603271 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df [sys.catalog]: This master's current role is: LEADER
I20260620 20:25:24.606061 13946 mvcc.cc:205] Tried to move back new op lower bound from 7299019261881393152 to 7299019261701382144. Current Snapshot: MvccSnapshot[applied={T|T < 7299019261881393152}]
I20260620 20:25:24.619561 13929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f9e04238bf42421ba89556e550f867df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de140a4b29724e76ab6bfad839e43409" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 33343 } } peers { permanent_uuid: "6227782cf4294bc2813c2f1ea0f4229e" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 36085 } } peers { permanent_uuid: "f9e04238bf42421ba89556e550f867df" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 45817 } } }
I20260620 20:25:24.619850 13929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:25:24.637072 13944 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:25:24.639667 13944 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:25:24.654737 13944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9e04238bf42421ba89556e550f867df: Generated new TSK 0
I20260620 20:25:24.655907 13944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:25:25.186980 13926 catalog_manager.cc:1259] Loaded cluster ID: 92ec327c445b4790b618e8df7be9b8a3
I20260620 20:25:25.187078 13926 catalog_manager.cc:1552] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409: loading cluster ID for follower catalog manager: success
I20260620 20:25:25.189406 13926 catalog_manager.cc:1574] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409: acquiring CA information for follower catalog manager: success
I20260620 20:25:25.191198 13926 catalog_manager.cc:1602] T 00000000000000000000000000000000 P de140a4b29724e76ab6bfad839e43409: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:25:25.310745 13940 catalog_manager.cc:1259] Loaded cluster ID: 92ec327c445b4790b618e8df7be9b8a3
I20260620 20:25:25.310853 13940 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e: loading cluster ID for follower catalog manager: success
I20260620 20:25:25.313408 13940 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e: acquiring CA information for follower catalog manager: success
I20260620 20:25:25.315218 13940 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6227782cf4294bc2813c2f1ea0f4229e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:25:25.743337 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 13670
I20260620 20:25:25.744645 13733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:25.853811 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 13670
I20260620 20:25:25.874724 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 13745
I20260620 20:25:25.876106 13808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:25.987887 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 13745
I20260620 20:25:26.008486 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 13820
I20260620 20:25:26.009593 13882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:26.117471 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 13820
2026-06-20T20:25:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4979 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:25:26 dist-test-slave-zzwf krb5kdc[13963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:25:26 dist-test-slave-zzwf krb5kdc[13963](info): set up 2 sockets
Jun 20 20:25:26 dist-test-slave-zzwf krb5kdc[13963](info): commencing operation
krb5kdc: starting...
W20260620 20:25:27.230410 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:25:27 dist-test-slave-zzwf krb5kdc[13963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:25:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:25:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:25:27.469233 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:44803
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:35119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:44803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:25:27.794400 13976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:27.794701 13976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:27.794875 13976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:27.802217 13976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:25:27.802426 13976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:27.802529 13976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:27.802631 13976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:25:27.802711 13976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:25:27.812907 13976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:35119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:44803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:44803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:27.814302 13976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:27.815577 13976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:27.828251 13981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:27.828230 13984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:27.828291 13982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:27.829216 13976 server_base.cc:1034] running on GCE node
I20260620 20:25:27.830006 13976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:27.831460 13976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:27.832671 13976 hybrid_clock.cc:648] HybridClock initialized: now 1781987127832622 us; error 43 us; skew 500 ppm
I20260620 20:25:27.833027 13976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:27 dist-test-slave-zzwf krb5kdc[13963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987127, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:27.836169 13976 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:25:27.837781 13976 webserver.cc:460] Webserver started at http://127.31.235.190:43225/ using document root <none> and password file <none>
I20260620 20:25:27.838488 13976 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:27.838627 13976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:27.838932 13976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:25:27.842149 13976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "b4de7b85e5194070884f38e4a047117f"
format_stamp: "Formatted at 2026-06-20 20:25:27 on dist-test-slave-zzwf"
server_key: "7ebd73b7c8a017c0540350e292a564a9"
server_key_iv: "f3428bb6cbb0fcba870cbcc8ccd2f151"
server_key_version: "encryptionkey@0"
I20260620 20:25:27.842947 13976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "b4de7b85e5194070884f38e4a047117f"
format_stamp: "Formatted at 2026-06-20 20:25:27 on dist-test-slave-zzwf"
server_key: "7ebd73b7c8a017c0540350e292a564a9"
server_key_iv: "f3428bb6cbb0fcba870cbcc8ccd2f151"
server_key_version: "encryptionkey@0"
I20260620 20:25:27.848340 13976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:25:27.852315 13991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:27.853858 13976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:25:27.854058 13976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "b4de7b85e5194070884f38e4a047117f"
format_stamp: "Formatted at 2026-06-20 20:25:27 on dist-test-slave-zzwf"
server_key: "7ebd73b7c8a017c0540350e292a564a9"
server_key_iv: "f3428bb6cbb0fcba870cbcc8ccd2f151"
server_key_version: "encryptionkey@0"
I20260620 20:25:27.854274 13976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:27.869902 13976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:27.875001 13976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:27.875361 13976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:27.896448 13976 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:44803
I20260620 20:25:27.896492 14042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:44803 every 8 connection(s)
I20260620 20:25:27.898046 13976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:25:27.901970 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:25:27.901995 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 13976
I20260620 20:25:27.902236 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:25:27.902971 32686 external_mini_cluster.cc:1428] Setting key 5497599de28a3dea7e297ac8b88f4e83
I20260620 20:25:27.916456 14043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f: Bootstrap starting.
Jun 20 20:25:27 dist-test-slave-zzwf krb5kdc[13963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:25:27.920567 14043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f: Neither blocks nor log segments found. Creating new log.
I20260620 20:25:27.921844 14043 log.cc:824] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:25:27.925307 14043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f: No bootstrap required, opened a new log
I20260620 20:25:27.928956 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:27.904815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43470 (local address 127.31.235.190:44803)
0620 20:25:27.905775 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:27.905788 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:27.905836 (+ 48us) server_negotiation.cc:407] Connection header received
0620 20:25:27.908372 (+ 2536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:27.908398 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:27.908864 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:27.909329 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:27.911242 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:27.912701 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:27.914723 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:27.915575 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:27.915738 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:27.920862 (+ 5124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:27.920871 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:27.920888 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:27.920958 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:27.924540 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:27.925338 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:27.925345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:27.925351 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:27.925619 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:27.926075 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:27.926079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:27.926081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:27.926641 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:27.926959 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:27.927446 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 20:25:27.927893 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":439,"threads_started":1}
I20260620 20:25:27.932920 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:25:27.934420 14043 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4de7b85e5194070884f38e4a047117f" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 44803 } }
I20260620 20:25:27.934854 14043 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:25:27.934974 14043 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4de7b85e5194070884f38e4a047117f, State: Initialized, Role: FOLLOWER
I20260620 20:25:27.935760 14043 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4de7b85e5194070884f38e4a047117f" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 44803 } }
I20260620 20:25:27.936059 14043 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:25:27.936239 14043 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:25:27.936443 14043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:25:27.938973 14043 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4de7b85e5194070884f38e4a047117f" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 44803 } }
I20260620 20:25:27.939553 14043 leader_election.cc:302] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4de7b85e5194070884f38e4a047117f; no voters:
I20260620 20:25:27.940218 14043 leader_election.cc:288] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:25:27.940390 14048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:25:27.941241 14048 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f [term 1 LEADER]: Becoming Leader. State: Replica: b4de7b85e5194070884f38e4a047117f, State: Running, Role: LEADER
I20260620 20:25:27.942298 14048 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4de7b85e5194070884f38e4a047117f" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 44803 } }
I20260620 20:25:27.942827 14043 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:25:27.947732 14049 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4de7b85e5194070884f38e4a047117f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4de7b85e5194070884f38e4a047117f" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 44803 } } }
I20260620 20:25:27.948096 14049 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f [sys.catalog]: This master's current role is: LEADER
I20260620 20:25:27.948586 14050 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4de7b85e5194070884f38e4a047117f. Latest consensus state: current_term: 1 leader_uuid: "b4de7b85e5194070884f38e4a047117f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4de7b85e5194070884f38e4a047117f" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 44803 } } }
I20260620 20:25:27.948905 14050 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f [sys.catalog]: This master's current role is: LEADER
I20260620 20:25:27.949108 14056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:25:27.955173 14056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:25:27.965075 14056 catalog_manager.cc:1347] Generated new cluster ID: 36b9d6433be4408c8ade87a14eafe91c
I20260620 20:25:27.965180 14056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:25:27.974176 14056 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:25:27.976058 14056 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:25:27.987445 14056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4de7b85e5194070884f38e4a047117f: Generated new TSK 0
I20260620 20:25:27.988516 14056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:25:28.045604 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 13976
I20260620 20:25:28.046788 14038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:28.144603 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 13976
2026-06-20T20:25:28Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2022 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:25:28 dist-test-slave-zzwf krb5kdc[14072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:25:28 dist-test-slave-zzwf krb5kdc[14072](info): set up 2 sockets
Jun 20 20:25:28 dist-test-slave-zzwf krb5kdc[14072](info): commencing operation
krb5kdc: starting...
W20260620 20:25:29.266812 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:25:29 dist-test-slave-zzwf krb5kdc[14072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:25:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:25:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:25:29.509886 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:46879
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:46879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:25:29.852111 14085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:29.852414 14085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:29.852612 14085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:29.860195 14085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:25:29.860330 14085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:29.860455 14085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:29.860576 14085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:25:29.860654 14085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:25:29.871106 14085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:46879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:46879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:29.872435 14085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:29.873762 14085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:29.886180 14090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:29.886111 14093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:29.886204 14091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:29.887324 14085 server_base.cc:1034] running on GCE node
I20260620 20:25:29.888104 14085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:29.889719 14085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:29.890954 14085 hybrid_clock.cc:648] HybridClock initialized: now 1781987129890900 us; error 47 us; skew 500 ppm
I20260620 20:25:29.891347 14085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:29 dist-test-slave-zzwf krb5kdc[14072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987129, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:29.894714 14085 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:25:29.896416 14085 webserver.cc:460] Webserver started at http://127.31.235.190:36109/ using document root <none> and password file <none>
I20260620 20:25:29.897112 14085 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:29.897255 14085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:29.897606 14085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:25:29.900651 14085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "e7e13fdf76554383b4738bd4829178a1"
format_stamp: "Formatted at 2026-06-20 20:25:29 on dist-test-slave-zzwf"
server_key: "5e5296ec6efcf2a56479f78a7c74f25a"
server_key_iv: "cd682919ffc98391461ff3542ceb7a01"
server_key_version: "encryptionkey@0"
I20260620 20:25:29.901409 14085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "e7e13fdf76554383b4738bd4829178a1"
format_stamp: "Formatted at 2026-06-20 20:25:29 on dist-test-slave-zzwf"
server_key: "5e5296ec6efcf2a56479f78a7c74f25a"
server_key_iv: "cd682919ffc98391461ff3542ceb7a01"
server_key_version: "encryptionkey@0"
I20260620 20:25:29.906436 14085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:25:29.910050 14100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:29.911484 14085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 20:25:29.911661 14085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "e7e13fdf76554383b4738bd4829178a1"
format_stamp: "Formatted at 2026-06-20 20:25:29 on dist-test-slave-zzwf"
server_key: "5e5296ec6efcf2a56479f78a7c74f25a"
server_key_iv: "cd682919ffc98391461ff3542ceb7a01"
server_key_version: "encryptionkey@0"
I20260620 20:25:29.911857 14085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:29.936596 14085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:29.941008 14085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:29.941318 14085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:29.961112 14085 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:46879
I20260620 20:25:29.961159 14151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:46879 every 8 connection(s)
I20260620 20:25:29.962492 14085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:25:29.966430 14152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:25:29.972918 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 14085
I20260620 20:25:29.973210 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:25:29.973969 32686 external_mini_cluster.cc:1428] Setting key 7478bcc644d6d88f4e53dda0565ed870
I20260620 20:25:29.978226 14152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1: Bootstrap starting.
I20260620 20:25:29.981666 14152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1: Neither blocks nor log segments found. Creating new log.
I20260620 20:25:29.982795 14152 log.cc:824] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1: Log is configured to *not* fsync() on all Append() calls
I20260620 20:25:29.985721 14152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1: No bootstrap required, opened a new log
Jun 20 20:25:29 dist-test-slave-zzwf krb5kdc[14072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:25:29.994817 14152 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e13fdf76554383b4738bd4829178a1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46879 } }
I20260620 20:25:29.995225 14152 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:25:29.995337 14152 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e13fdf76554383b4738bd4829178a1, State: Initialized, Role: FOLLOWER
I20260620 20:25:29.996026 14152 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e13fdf76554383b4738bd4829178a1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46879 } }
I20260620 20:25:29.996274 14152 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:25:29.996455 14152 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:25:29.996661 14152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:25:29.999202 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:29.976145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49766 (local address 127.31.235.190:46879)
0620 20:25:29.976983 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:29.976998 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:29.977052 (+ 54us) server_negotiation.cc:407] Connection header received
0620 20:25:29.978986 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:29.979012 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:29.979403 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:29.979832 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:29.981734 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:29.983387 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:29.985064 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:29.985948 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:29.986187 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:29.991093 (+ 4906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:29.991105 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:29.991127 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:29.991175 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:29.994841 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:29.995719 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:29.995726 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:29.995732 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:29.995900 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:29.996322 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:29.996327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:29.996330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:29.996974 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:29.997320 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:29.997746 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 20:25:29.998195 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":381,"threads_started":1}
I20260620 20:25:29.999487 14152 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e13fdf76554383b4738bd4829178a1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46879 } }
I20260620 20:25:30.000140 14152 leader_election.cc:302] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7e13fdf76554383b4738bd4829178a1; no voters:
I20260620 20:25:30.000861 14152 leader_election.cc:288] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:25:30.001001 14157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:25:30.003140 14152 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:25:30.003379 14157 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 1 LEADER]: Becoming Leader. State: Replica: e7e13fdf76554383b4738bd4829178a1, State: Running, Role: LEADER
I20260620 20:25:30.004138 14157 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e13fdf76554383b4738bd4829178a1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46879 } }
I20260620 20:25:30.011303 14158 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7e13fdf76554383b4738bd4829178a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e13fdf76554383b4738bd4829178a1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46879 } } }
I20260620 20:25:30.011353 14159 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7e13fdf76554383b4738bd4829178a1. Latest consensus state: current_term: 1 leader_uuid: "e7e13fdf76554383b4738bd4829178a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e13fdf76554383b4738bd4829178a1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46879 } } }
I20260620 20:25:30.011797 14158 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:25:30.011797 14159 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:25:30.012719 14166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:25:30.018306 14166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:25:30.027068 14166 catalog_manager.cc:1347] Generated new cluster ID: ce240b7860644902997330036a94842f
I20260620 20:25:30.027220 14166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:25:30.039425 14166 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:25:30.041329 14166 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:25:30.050369 14166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1: Generated new TSK 0
I20260620 20:25:30.051395 14166 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
I20260620 20:25:30.118237 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:46879
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:25:30.466434 14178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:30.466724 14178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:30.466822 14178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:30.474160 14178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:30.474277 14178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:30.474395 14178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:25:30.484160 14178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:46879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:30.485460 14178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:30.486831 14178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:30.502063 14184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:30.502031 14183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:30.503235 14178 server_base.cc:1034] running on GCE node
W20260620 20:25:30.503300 14186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:30.503947 14178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:30.504982 14178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:30.506188 14178 hybrid_clock.cc:648] HybridClock initialized: now 1781987130506148 us; error 34 us; skew 500 ppm
I20260620 20:25:30.506532 14178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:30 dist-test-slave-zzwf krb5kdc[14072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987130, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:30.509701 14178 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:25:30.511153 14178 webserver.cc:460] Webserver started at http://127.31.235.129:45965/ using document root <none> and password file <none>
I20260620 20:25:30.511883 14178 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:30.512027 14178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:30.512354 14178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:25:30.515195 14178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "cb8a664213f642c890df98ac673d6116"
format_stamp: "Formatted at 2026-06-20 20:25:30 on dist-test-slave-zzwf"
server_key: "5f330f659ddf70e14d96c17765550276"
server_key_iv: "8465c82177519e1ea9301d88b1131532"
server_key_version: "encryptionkey@0"
I20260620 20:25:30.516045 14178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "cb8a664213f642c890df98ac673d6116"
format_stamp: "Formatted at 2026-06-20 20:25:30 on dist-test-slave-zzwf"
server_key: "5f330f659ddf70e14d96c17765550276"
server_key_iv: "8465c82177519e1ea9301d88b1131532"
server_key_version: "encryptionkey@0"
I20260620 20:25:30.521274 14178 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:25:30.524893 14193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:30.526324 14178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:25:30.526492 14178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "cb8a664213f642c890df98ac673d6116"
format_stamp: "Formatted at 2026-06-20 20:25:30 on dist-test-slave-zzwf"
server_key: "5f330f659ddf70e14d96c17765550276"
server_key_iv: "8465c82177519e1ea9301d88b1131532"
server_key_version: "encryptionkey@0"
I20260620 20:25:30.526707 14178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:30.540944 14178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:30.545050 14178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:30.545356 14178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:30.546342 14178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:25:30.547646 14178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:25:30.547744 14178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:30.547881 14178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:25:30.547953 14178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:30.580708 14178 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:32811
I20260620 20:25:30.580790 14306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:32811 every 8 connection(s)
I20260620 20:25:30.582002 14178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:25:30.583175 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 14178
I20260620 20:25:30.583410 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:25:30.584303 32686 external_mini_cluster.cc:1428] Setting key 7519254fb7f55acb67bceb5d4f7f285c
Jun 20 20:25:30 dist-test-slave-zzwf krb5kdc[14072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987130, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:25:30.605033 14309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:30.584955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:35037 (local address 127.31.235.190:46879)
0620 20:25:30.585575 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:30.585581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:30.586658 (+ 1077us) server_negotiation.cc:407] Connection header received
0620 20:25:30.588733 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:30.588739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:30.588830 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:30.588952 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:30.590568 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:30.591364 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:30.593410 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:30.593904 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:30.594009 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:30.598795 (+ 4786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:30.598800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:30.598803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:30.598835 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:30.601385 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:30.602424 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:30.602428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:30.602430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:30.602548 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:30.603106 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:30.603109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:30.603112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:30.603375 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:30.603524 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:30.604525 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0620 20:25:30.604756 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":308,"threads_started":1}
I20260620 20:25:30.606873 14310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:30.585567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.129:35037)
0620 20:25:30.586483 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:30.586543 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:30.588442 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:30.589205 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:30.589214 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:30.589798 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:30.590384 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:30.590402 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:30.591522 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:30.591534 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:30.593222 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:30.593239 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:30.594156 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:30.594166 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:30.594528 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:30.595346 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:30.595383 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:30.598573 (+ 3190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:30.601556 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:30.601570 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:30.601592 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:30.602299 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:30.602668 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:30.602677 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:30.602685 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:30.602952 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:30.603524 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:30.603536 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:30.604051 (+ 515us) client_negotiation.cc:769] Sending connection context
0620 20:25:30.604422 (+ 371us) client_negotiation.cc:240] Negotiation successful
0620 20:25:30.605098 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":302,"threads_started":1}
I20260620 20:25:30.608634 14307 heartbeater.cc:347] Connected to a master server at 127.31.235.190:46879
I20260620 20:25:30.608955 14307 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.31.235.130@KRBTEST.COM; defaulting to no policy
I20260620 20:25:30.609797 14307 heartbeater.cc:511] Master 127.31.235.190:46879 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.130@KRBTEST.COM" created.
I20260620 20:25:30.612365 14117 ts_manager.cc:194] Registered new tserver with Master: cb8a664213f642c890df98ac673d6116 (127.31.235.129:32811)
I20260620 20:25:30.614013 14117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:35037
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
I20260620 20:25:30.640712 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:46879
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:25:30.967430 14313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:30.967792 14313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:30.967927 14313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:30.979542 14313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:30.979684 14313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:30.979867 14313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:25:30.991174 14313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:46879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:30.992439 14313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:30.993772 14313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:31.006510 14318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:31.006487 14321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:31.006739 14319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:31.007436 14313 server_base.cc:1034] running on GCE node
I20260620 20:25:31.008101 14313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:31.009084 14313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:31.010339 14313 hybrid_clock.cc:648] HybridClock initialized: now 1781987131010275 us; error 62 us; skew 500 ppm
I20260620 20:25:31.010690 14313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:31 dist-test-slave-zzwf krb5kdc[14072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987131, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:31.014008 14313 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:25:31.015437 14313 webserver.cc:460] Webserver started at http://127.31.235.130:35081/ using document root <none> and password file <none>
I20260620 20:25:31.016137 14313 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:31.016275 14313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:31.016580 14313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:25:31.019455 14313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "2ce6988a818c4a8fb404023cf7ad39d1"
format_stamp: "Formatted at 2026-06-20 20:25:31 on dist-test-slave-zzwf"
server_key: "c08546fe3e1abc6ae55ea225d7843c2e"
server_key_iv: "e983f8375d46a1c972499f0b8098c097"
server_key_version: "encryptionkey@0"
I20260620 20:25:31.020215 14313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "2ce6988a818c4a8fb404023cf7ad39d1"
format_stamp: "Formatted at 2026-06-20 20:25:31 on dist-test-slave-zzwf"
server_key: "c08546fe3e1abc6ae55ea225d7843c2e"
server_key_iv: "e983f8375d46a1c972499f0b8098c097"
server_key_version: "encryptionkey@0"
I20260620 20:25:31.025622 14313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 20:25:31.029112 14328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:31.030742 14313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:25:31.030906 14313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "2ce6988a818c4a8fb404023cf7ad39d1"
format_stamp: "Formatted at 2026-06-20 20:25:31 on dist-test-slave-zzwf"
server_key: "c08546fe3e1abc6ae55ea225d7843c2e"
server_key_iv: "e983f8375d46a1c972499f0b8098c097"
server_key_version: "encryptionkey@0"
I20260620 20:25:31.031155 14313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:31.060932 14313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:31.065366 14313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:31.065706 14313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:31.066697 14313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:25:31.068101 14313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:25:31.068203 14313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:31.068349 14313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:25:31.068427 14313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:31.102406 14313 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:35781
I20260620 20:25:31.102454 14441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:35781 every 8 connection(s)
I20260620 20:25:31.103856 14313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:25:31.113864 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 14313
I20260620 20:25:31.114082 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:25:31.114666 32686 external_mini_cluster.cc:1428] Setting key eaaf6cd414309640cf74880ffdae1604
Jun 20 20:25:31 dist-test-slave-zzwf krb5kdc[14072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987131, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:25:31.126168 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:31.106629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:35083 (local address 127.31.235.190:46879)
0620 20:25:31.107037 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:31.107041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:31.108087 (+ 1046us) server_negotiation.cc:407] Connection header received
0620 20:25:31.109956 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:31.109961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:31.110031 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:31.110133 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:31.111714 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.112487 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.114108 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.114760 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.114893 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:31.118864 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:31.118873 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:31.118880 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:31.118925 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:31.122749 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:31.123525 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:31.123533 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:31.123539 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:31.123711 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:31.124185 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:31.124192 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:31.124198 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:31.124608 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:31.124762 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:31.125658 (+ 896us) server_negotiation.cc:299] Negotiation successful
0620 20:25:31.125870 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":195,"threads_started":1}
I20260620 20:25:31.127640 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:31.107181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.130:35083)
0620 20:25:31.107892 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:31.107944 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:31.109702 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:31.110360 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:31.110368 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:31.110927 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:31.111495 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:31.111513 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.112636 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.112640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:31.113947 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:31.113959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.115049 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.115057 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:31.115285 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:31.116129 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:31.116159 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:31.118623 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:31.122921 (+ 4298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:31.122930 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:31.122945 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:31.123383 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:31.123841 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:31.123845 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:31.123848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:31.124057 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:31.124725 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:31.124731 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:31.125143 (+ 412us) client_negotiation.cc:769] Sending connection context
0620 20:25:31.125463 (+ 320us) client_negotiation.cc:240] Negotiation successful
0620 20:25:31.126155 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":260,"threads_started":1}
I20260620 20:25:31.129122 14442 heartbeater.cc:347] Connected to a master server at 127.31.235.190:46879
I20260620 20:25:31.129436 14442 heartbeater.cc:464] Registering TS with master...
I20260620 20:25:31.130273 14442 heartbeater.cc:511] Master 127.31.235.190:46879 requested a full tablet report, sending...
I20260620 20:25:31.131979 14117 ts_manager.cc:194] Registered new tserver with Master: 2ce6988a818c4a8fb404023cf7ad39d1 (127.31.235.130:35781)
I20260620 20:25:31.132951 14117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:35083
WARNING: no policy specified for kudu/127.31.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
I20260620 20:25:31.169745 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:46879
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:25:31.492655 14448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:31.492925 14448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:31.493076 14448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:31.500494 14448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:31.500607 14448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:31.500802 14448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:25:31.510646 14448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:46879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:31.512017 14448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:31.513346 14448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:31.527307 14456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:31.527367 14453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:31.527403 14454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:31.528854 14448 server_base.cc:1034] running on GCE node
I20260620 20:25:31.529650 14448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:31.530714 14448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:31.531946 14448 hybrid_clock.cc:648] HybridClock initialized: now 1781987131531868 us; error 77 us; skew 500 ppm
I20260620 20:25:31.532286 14448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:31 dist-test-slave-zzwf krb5kdc[14072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987131, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:31.535699 14448 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:25:31.537261 14448 webserver.cc:460] Webserver started at http://127.31.235.131:32775/ using document root <none> and password file <none>
I20260620 20:25:31.538172 14448 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:31.538313 14448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:31.538643 14448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:25:31.541868 14448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "6e3716d2910a408d9793c2dd8281cebd"
format_stamp: "Formatted at 2026-06-20 20:25:31 on dist-test-slave-zzwf"
server_key: "247e4ba33aa47d599e632b51cb92b96d"
server_key_iv: "a0f021742008d264aaf3968d7f5b0057"
server_key_version: "encryptionkey@0"
I20260620 20:25:31.542713 14448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "6e3716d2910a408d9793c2dd8281cebd"
format_stamp: "Formatted at 2026-06-20 20:25:31 on dist-test-slave-zzwf"
server_key: "247e4ba33aa47d599e632b51cb92b96d"
server_key_iv: "a0f021742008d264aaf3968d7f5b0057"
server_key_version: "encryptionkey@0"
I20260620 20:25:31.548164 14448 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:25:31.552012 14463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:31.553632 14448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:25:31.553805 14448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "6e3716d2910a408d9793c2dd8281cebd"
format_stamp: "Formatted at 2026-06-20 20:25:31 on dist-test-slave-zzwf"
server_key: "247e4ba33aa47d599e632b51cb92b96d"
server_key_iv: "a0f021742008d264aaf3968d7f5b0057"
server_key_version: "encryptionkey@0"
I20260620 20:25:31.554001 14448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:31.570699 14448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:31.575191 14448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:31.575491 14448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:31.576514 14448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:25:31.577948 14448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:25:31.578023 14448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:31.578145 14448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:25:31.578207 14448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:31.616567 14448 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:42689
I20260620 20:25:31.616653 14576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:42689 every 8 connection(s)
I20260620 20:25:31.617758 14307 heartbeater.cc:503] Master 127.31.235.190:46879 was elected leader, sending a full tablet report...
I20260620 20:25:31.618007 14448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:25:31.623320 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 14448
I20260620 20:25:31.623521 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:25:31.623936 32686 external_mini_cluster.cc:1428] Setting key 0e546189108e5773b449017be1b89347
Jun 20 20:25:31 dist-test-slave-zzwf krb5kdc[14072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987131, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:25:31.640885 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:31.621009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:44625 (local address 127.31.235.190:46879)
0620 20:25:31.621224 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:31.621231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:31.622495 (+ 1264us) server_negotiation.cc:407] Connection header received
0620 20:25:31.624642 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:31.624649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:31.624736 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:31.624877 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:31.627080 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.628059 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.629861 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.630322 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.630519 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:31.634293 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:31.634298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:31.634302 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:31.634338 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:31.637363 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:31.638185 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:31.638190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:31.638192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:31.638314 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:31.638836 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:31.638844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:31.638849 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:31.639246 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:31.639399 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:31.640434 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0620 20:25:31.640640 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 20:25:31.641932 14579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:31.621515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.131:44625)
0620 20:25:31.622323 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:31.622371 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:31.624305 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:31.625203 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:31.625218 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:31.625860 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:31.626875 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:31.626903 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.628239 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.628243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:31.629702 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:31.629714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.630610 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.630615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:31.630888 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:31.631583 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:31.631608 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:31.634021 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:31.637558 (+ 3537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:31.637567 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:31.637594 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:31.638040 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:31.638424 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:31.638429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:31.638433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:31.638684 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:31.639422 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:31.639434 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:31.639880 (+ 446us) client_negotiation.cc:769] Sending connection context
0620 20:25:31.640265 (+ 385us) client_negotiation.cc:240] Negotiation successful
0620 20:25:31.640704 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":298,"threads_started":1}
I20260620 20:25:31.643224 14577 heartbeater.cc:347] Connected to a master server at 127.31.235.190:46879
I20260620 20:25:31.643545 14577 heartbeater.cc:464] Registering TS with master...
I20260620 20:25:31.644301 14577 heartbeater.cc:511] Master 127.31.235.190:46879 requested a full tablet report, sending...
I20260620 20:25:31.645862 14117 ts_manager.cc:194] Registered new tserver with Master: 6e3716d2910a408d9793c2dd8281cebd (127.31.235.131:42689)
I20260620 20:25:31.646656 14117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:44625
I20260620 20:25:31.653168 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:25:31.671725 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:31.658074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49774 (local address 127.31.235.190:46879)
0620 20:25:31.658266 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:31.658270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:31.658710 (+ 440us) server_negotiation.cc:407] Connection header received
0620 20:25:31.658853 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:31.658858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:31.658941 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:31.659069 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:31.660419 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.661308 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.662771 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.663298 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.663411 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:31.665992 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:31.666043 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:31.666046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:31.666089 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:31.668954 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:31.669744 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:31.669749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:31.669752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:31.669888 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:31.670268 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:31.670272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:31.670284 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:31.670632 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:31.670856 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:31.671281 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 20:25:31.671463 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:25:31.676302 14117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:25:31.683429 14117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:25:31.709009 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:31.696343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38462 (local address 127.31.235.131:42689)
0620 20:25:31.697284 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:31.697295 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:31.697886 (+ 591us) server_negotiation.cc:407] Connection header received
0620 20:25:31.698222 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:31.698231 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:31.698496 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:31.698680 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:31.699868 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.702063 (+ 2195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.706262 (+ 4199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.707740 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.707955 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:31.708165 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:31.708453 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 20:25:31.708626 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":578,"threads_started":1}
I20260620 20:25:31.709009 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:31.696109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:42689 (local address 127.0.0.1:38462)
0620 20:25:31.697732 (+ 1623us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:31.697777 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:31.697965 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:31.699073 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:31.699082 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:31.699120 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:31.699622 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:31.699641 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.702298 (+ 2657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.702307 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:31.705971 (+ 3664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:31.705995 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.707943 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.707951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:31.708173 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:31.708237 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 20:25:31.708412 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 20:25:31.708637 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1305,"thread_start_us":1375,"threads_started":1}
I20260620 20:25:31.713958 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:31.697631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44238 (local address 127.31.235.129:32811)
0620 20:25:31.706649 (+ 9018us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:31.706660 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:31.706695 (+ 35us) server_negotiation.cc:407] Connection header received
0620 20:25:31.706797 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:31.706805 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:31.707106 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:31.707328 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:31.707984 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.709632 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.711786 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.712883 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.713097 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:31.713278 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:31.713454 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 20:25:31.713637 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8819,"thread_start_us":8657,"threads_started":1}
I20260620 20:25:31.714012 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:31.696912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:32811 (local address 127.0.0.1:44238)
0620 20:25:31.697732 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:31.697777 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:31.697964 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:31.707330 (+ 9366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:31.707335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:31.707365 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:31.707772 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:31.707786 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.709793 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.709798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:31.711629 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:31.711650 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.713093 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.713099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:31.713259 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:31.713294 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 20:25:31.713412 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 20:25:31.713581 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":24,"thread_start_us":1205,"threads_started":1}
I20260620 20:25:31.713582 14511 tablet_service.cc:1467] Processing CreateTablet for tablet f781d967d641429595bf5ee4f9ff36a7 (DEFAULT_TABLE table=test-table [id=1917a807142049d998cf6ad4da3c70c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:25:31.715425 14511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f781d967d641429595bf5ee4f9ff36a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:25:31.715998 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:31.698283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:35781 (local address 127.0.0.1:53160)
0620 20:25:31.707305 (+ 9022us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:31.707325 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:31.707445 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:31.708257 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:31.708261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:31.708288 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:31.708716 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:31.708729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.710717 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.710725 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:31.712874 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:31.712889 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.714960 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.714966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:31.715221 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:31.715258 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 20:25:31.715372 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 20:25:31.715552 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8880,"mutex_wait_us":1,"thread_start_us":9270,"threads_started":1}
I20260620 20:25:31.716913 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:31.698555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53160 (local address 127.31.235.130:35781)
0620 20:25:31.706894 (+ 8339us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:31.706902 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:31.707577 (+ 675us) server_negotiation.cc:407] Connection header received
0620 20:25:31.707682 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:31.707688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:31.707902 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:31.708095 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:31.708926 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.710593 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.713060 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.714426 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.714594 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:31.714788 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:31.716374 (+ 1586us) server_negotiation.cc:299] Negotiation successful
0620 20:25:31.716568 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8198,"thread_start_us":8091,"threads_started":1}
I20260620 20:25:31.717674 14242 tablet_service.cc:1467] Processing CreateTablet for tablet f781d967d641429595bf5ee4f9ff36a7 (DEFAULT_TABLE table=test-table [id=1917a807142049d998cf6ad4da3c70c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:25:31.719187 14242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f781d967d641429595bf5ee4f9ff36a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:25:31.720551 14376 tablet_service.cc:1467] Processing CreateTablet for tablet f781d967d641429595bf5ee4f9ff36a7 (DEFAULT_TABLE table=test-table [id=1917a807142049d998cf6ad4da3c70c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:25:31.722322 14376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f781d967d641429595bf5ee4f9ff36a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:25:31.729198 14595 tablet_bootstrap.cc:492] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd: Bootstrap starting.
I20260620 20:25:31.732875 14595 tablet_bootstrap.cc:654] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd: Neither blocks nor log segments found. Creating new log.
I20260620 20:25:31.733227 14596 tablet_bootstrap.cc:492] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116: Bootstrap starting.
I20260620 20:25:31.734277 14595 log.cc:824] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd: Log is configured to *not* fsync() on all Append() calls
I20260620 20:25:31.736652 14596 tablet_bootstrap.cc:654] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116: Neither blocks nor log segments found. Creating new log.
I20260620 20:25:31.737582 14595 tablet_bootstrap.cc:492] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd: No bootstrap required, opened a new log
I20260620 20:25:31.737841 14598 tablet_bootstrap.cc:492] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1: Bootstrap starting.
I20260620 20:25:31.737924 14595 ts_tablet_manager.cc:1397] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 20:25:31.738726 14596 log.cc:824] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116: Log is configured to *not* fsync() on all Append() calls
I20260620 20:25:31.741726 14598 tablet_bootstrap.cc:654] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1: Neither blocks nor log segments found. Creating new log.
I20260620 20:25:31.741909 14596 tablet_bootstrap.cc:492] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116: No bootstrap required, opened a new log
I20260620 20:25:31.742197 14596 ts_tablet_manager.cc:1397] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 20:25:31.742924 14598 log.cc:824] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1: Log is configured to *not* fsync() on all Append() calls
I20260620 20:25:31.745884 14598 tablet_bootstrap.cc:492] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1: No bootstrap required, opened a new log
I20260620 20:25:31.746143 14598 ts_tablet_manager.cc:1397] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260620 20:25:31.746196 14595 raft_consensus.cc:348] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 } } peers { permanent_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35781 } } peers { permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 } }
I20260620 20:25:31.746603 14595 raft_consensus.cc:374] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:25:31.746718 14595 raft_consensus.cc:729] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3716d2910a408d9793c2dd8281cebd, State: Initialized, Role: FOLLOWER
I20260620 20:25:31.747308 14595 consensus_queue.cc:260] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 } } peers { permanent_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35781 } } peers { permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 } }
I20260620 20:25:31.748433 14577 heartbeater.cc:503] Master 127.31.235.190:46879 was elected leader, sending a full tablet report...
I20260620 20:25:31.748889 14595 ts_tablet_manager.cc:1428] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260620 20:25:31.752532 14596 raft_consensus.cc:348] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 } } peers { permanent_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35781 } } peers { permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 } }
I20260620 20:25:31.753060 14596 raft_consensus.cc:374] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:25:31.753168 14596 raft_consensus.cc:729] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8a664213f642c890df98ac673d6116, State: Initialized, Role: FOLLOWER
I20260620 20:25:31.753813 14596 consensus_queue.cc:260] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 } } peers { permanent_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35781 } } peers { permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 } }
I20260620 20:25:31.756392 14596 ts_tablet_manager.cc:1428] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 20:25:31.756217 14598 raft_consensus.cc:348] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 } } peers { permanent_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35781 } } peers { permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 } }
I20260620 20:25:31.756646 14598 raft_consensus.cc:374] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:25:31.756750 14598 raft_consensus.cc:729] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce6988a818c4a8fb404023cf7ad39d1, State: Initialized, Role: FOLLOWER
I20260620 20:25:31.757361 14598 consensus_queue.cc:260] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 } } peers { permanent_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35781 } } peers { permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 } }
I20260620 20:25:31.758575 14442 heartbeater.cc:503] Master 127.31.235.190:46879 was elected leader, sending a full tablet report...
I20260620 20:25:31.759667 14598 ts_tablet_manager.cc:1428] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
W20260620 20:25:31.834474 14308 tablet.cc:2369] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:25:31.856133 14443 tablet.cc:2369] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:25:31.870487 14578 tablet.cc:2369] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:25:31.914002 14603 raft_consensus.cc:482] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:25:31.914332 14603 raft_consensus.cc:504] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 } } peers { permanent_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35781 } } peers { permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 } }
I20260620 20:25:31.916196 14603 leader_election.cc:288] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e3716d2910a408d9793c2dd8281cebd (127.31.235.131:42689), cb8a664213f642c890df98ac673d6116 (127.31.235.129:32811)
I20260620 20:25:31.925678 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:31.916553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:56687 (local address 127.31.235.131:42689)
0620 20:25:31.916777 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:31.916785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:31.917464 (+ 679us) server_negotiation.cc:407] Connection header received
0620 20:25:31.917839 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:31.917847 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:31.917929 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:31.918012 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:31.919207 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.920000 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.922927 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.924208 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.924367 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:31.924431 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:31.925220 (+ 789us) server_negotiation.cc:299] Negotiation successful
0620 20:25:31.925365 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:25:31.925679 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:31.917256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:47203 (local address 127.31.235.129:32811)
0620 20:25:31.917478 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:31.917511 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:31.917613 (+ 102us) server_negotiation.cc:407] Connection header received
0620 20:25:31.917867 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:31.917872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:31.917945 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:31.918011 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:31.919139 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.920427 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.922989 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.924221 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.924355 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:31.924431 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:31.925220 (+ 789us) server_negotiation.cc:299] Negotiation successful
0620 20:25:31.925365 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 20:25:31.926733 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:31.916659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:32811 (local address 127.31.235.130:47203)
0620 20:25:31.917518 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:31.917537 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:31.917705 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:31.918230 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:31.918236 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:31.918293 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:31.918905 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:31.918929 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.920571 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.920576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:31.922813 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:31.922827 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.924627 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.924631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:31.924911 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:31.925055 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 20:25:31.925877 (+ 822us) client_negotiation.cc:240] Negotiation successful
0620 20:25:31.926040 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":138,"spinlock_wait_cycles":4352,"thread_start_us":926,"threads_started":1}
I20260620 20:25:31.926762 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:31.916531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:42689 (local address 127.31.235.130:56687)
0620 20:25:31.917342 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:31.917364 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:31.917632 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:31.918230 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:31.918236 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:31.918293 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:31.919008 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:31.919022 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.920155 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.920162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:31.922672 (+ 2510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:31.922691 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:31.924624 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:31.924630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:31.924912 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:31.925054 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 20:25:31.925936 (+ 882us) client_negotiation.cc:240] Negotiation successful
0620 20:25:31.926067 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":196,"spinlock_wait_cycles":36736,"thread_start_us":393,"threads_started":1}
I20260620 20:25:31.927915 14531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f781d967d641429595bf5ee4f9ff36a7" candidate_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3716d2910a408d9793c2dd8281cebd" is_pre_election: true
I20260620 20:25:31.928458 14531 raft_consensus.cc:2454] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ce6988a818c4a8fb404023cf7ad39d1 in term 0.
I20260620 20:25:31.929275 14329 leader_election.cc:302] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ce6988a818c4a8fb404023cf7ad39d1, 6e3716d2910a408d9793c2dd8281cebd; no voters:
I20260620 20:25:31.929384 14262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f781d967d641429595bf5ee4f9ff36a7" candidate_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb8a664213f642c890df98ac673d6116" is_pre_election: true
I20260620 20:25:31.930047 14262 raft_consensus.cc:2454] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ce6988a818c4a8fb404023cf7ad39d1 in term 0.
I20260620 20:25:31.930107 14603 raft_consensus.cc:2781] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:25:31.930883 14603 raft_consensus.cc:482] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:25:31.931105 14603 raft_consensus.cc:3037] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:25:31.934947 14603 raft_consensus.cc:504] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 } } peers { permanent_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35781 } } peers { permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 } }
I20260620 20:25:31.935703 14603 leader_election.cc:288] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 [CANDIDATE]: Term 1 election: Requested vote from peers 6e3716d2910a408d9793c2dd8281cebd (127.31.235.131:42689), cb8a664213f642c890df98ac673d6116 (127.31.235.129:32811)
I20260620 20:25:31.936282 14531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f781d967d641429595bf5ee4f9ff36a7" candidate_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3716d2910a408d9793c2dd8281cebd"
I20260620 20:25:31.936533 14531 raft_consensus.cc:3037] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:25:31.936561 14262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f781d967d641429595bf5ee4f9ff36a7" candidate_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb8a664213f642c890df98ac673d6116"
I20260620 20:25:31.936755 14262 raft_consensus.cc:3037] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:25:31.939088 14531 raft_consensus.cc:2454] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ce6988a818c4a8fb404023cf7ad39d1 in term 1.
I20260620 20:25:31.939563 14329 leader_election.cc:302] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ce6988a818c4a8fb404023cf7ad39d1, 6e3716d2910a408d9793c2dd8281cebd; no voters:
I20260620 20:25:31.939874 14603 raft_consensus.cc:2781] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:25:31.940237 14262 raft_consensus.cc:2454] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ce6988a818c4a8fb404023cf7ad39d1 in term 1.
I20260620 20:25:31.941180 14603 raft_consensus.cc:686] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 1 LEADER]: Becoming Leader. State: Replica: 2ce6988a818c4a8fb404023cf7ad39d1, State: Running, Role: LEADER
I20260620 20:25:31.941969 14603 consensus_queue.cc:237] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 } } peers { permanent_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35781 } } peers { permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 } }
I20260620 20:25:31.947490 14116 catalog_manager.cc:5495] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ce6988a818c4a8fb404023cf7ad39d1 (127.31.235.130). New cstate: current_term: 1 leader_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 } health_report { overall_health: UNKNOWN } } }
I20260620 20:25:32.012176 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:32.006252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53168 (local address 127.31.235.130:35781)
0620 20:25:32.006431 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:32.006436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:32.006452 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:25:32.006546 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:32.006549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:32.006630 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:32.006756 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:25:32.007151 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:32.007868 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:32.009324 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:32.010092 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:32.010194 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:32.010833 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:25:32.011475 (+ 642us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:25:32.011599 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:32.011745 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 20:25:32.011820 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:25:32.020380 14531 raft_consensus.cc:1261] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd [term 1 FOLLOWER]: Refusing update from remote peer 2ce6988a818c4a8fb404023cf7ad39d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:25:32.020381 14262 raft_consensus.cc:1261] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116 [term 1 FOLLOWER]: Refusing update from remote peer 2ce6988a818c4a8fb404023cf7ad39d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:25:32.021308 14606 consensus_queue.cc:1035] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:25:32.021739 14603 consensus_queue.cc:1035] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:25:32.033587 14610 mvcc.cc:205] Tried to move back new op lower bound from 7299019292745785344 to 7299019292439359488. Current Snapshot: MvccSnapshot[applied={T|T < 7299019292745785344}]
I20260620 20:25:32.041949 14611 mvcc.cc:205] Tried to move back new op lower bound from 7299019292745785344 to 7299019292439359488. Current Snapshot: MvccSnapshot[applied={T|T < 7299019292745785344}]
I20260620 20:25:32.055068 14116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:25:32.055428 14116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:25:32.059085 14116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1: Sending DeleteTablet for 3 replicas of tablet f781d967d641429595bf5ee4f9ff36a7
I20260620 20:25:32.060216 14511 tablet_service.cc:1514] Processing DeleteTablet for tablet f781d967d641429595bf5ee4f9ff36a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:25:32 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:38462
I20260620 20:25:32.060379 14376 tablet_service.cc:1514] Processing DeleteTablet for tablet f781d967d641429595bf5ee4f9ff36a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:25:32 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:53160
I20260620 20:25:32.060889 14242 tablet_service.cc:1514] Processing DeleteTablet for tablet f781d967d641429595bf5ee4f9ff36a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:25:32 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:44238
I20260620 20:25:32.062186 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 14178
I20260620 20:25:32.071775 14621 tablet_replica.cc:330] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116: stopping tablet replica
I20260620 20:25:32.072585 14621 raft_consensus.cc:2229] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:25:32.073369 14621 raft_consensus.cc:2258] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:25:32.076377 14621 ts_tablet_manager.cc:1905] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:25:32.085963 14619 tablet_replica.cc:330] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd: stopping tablet replica
I20260620 20:25:32.086392 14619 raft_consensus.cc:2229] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:25:32.086874 14619 raft_consensus.cc:2258] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:25:32.087635 14620 tablet_replica.cc:330] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1: stopping tablet replica
I20260620 20:25:32.087983 14620 raft_consensus.cc:2229] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:25:32.088531 14620 raft_consensus.cc:2258] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:25:32.089426 14621 ts_tablet_manager.cc:1918] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:25:32.089630 14621 log.cc:1196] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/f781d967d641429595bf5ee4f9ff36a7
I20260620 20:25:32.090166 14621 ts_tablet_manager.cc:1939] T f781d967d641429595bf5ee4f9ff36a7 P cb8a664213f642c890df98ac673d6116: Deleting consensus metadata
I20260620 20:25:32.090767 14620 ts_tablet_manager.cc:1905] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:25:32.090773 14619 ts_tablet_manager.cc:1905] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:25:32.091984 14102 catalog_manager.cc:4841] TS cb8a664213f642c890df98ac673d6116 (127.31.235.129:32811): tablet f781d967d641429595bf5ee4f9ff36a7 (table test-table [id=1917a807142049d998cf6ad4da3c70c0]) successfully deleted
I20260620 20:25:32.095873 14619 ts_tablet_manager.cc:1918] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:25:32.096057 14619 log.cc:1196] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/f781d967d641429595bf5ee4f9ff36a7
I20260620 20:25:32.096560 14619 ts_tablet_manager.cc:1939] T f781d967d641429595bf5ee4f9ff36a7 P 6e3716d2910a408d9793c2dd8281cebd: Deleting consensus metadata
I20260620 20:25:32.098302 14620 ts_tablet_manager.cc:1918] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:25:32.098464 14620 log.cc:1196] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/f781d967d641429595bf5ee4f9ff36a7
I20260620 20:25:32.098865 14101 catalog_manager.cc:4841] TS 6e3716d2910a408d9793c2dd8281cebd (127.31.235.131:42689): tablet f781d967d641429595bf5ee4f9ff36a7 (table test-table [id=1917a807142049d998cf6ad4da3c70c0]) successfully deleted
I20260620 20:25:32.099632 14620 ts_tablet_manager.cc:1939] T f781d967d641429595bf5ee4f9ff36a7 P 2ce6988a818c4a8fb404023cf7ad39d1: Deleting consensus metadata
I20260620 20:25:32.101439 14101 catalog_manager.cc:4841] TS 2ce6988a818c4a8fb404023cf7ad39d1 (127.31.235.130:35781): tablet f781d967d641429595bf5ee4f9ff36a7 (table test-table [id=1917a807142049d998cf6ad4da3c70c0]) successfully deleted
Jun 20 20:25:32 dist-test-slave-zzwf krb5kdc[14072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:25:32.109995 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:32.063530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44250 (local address 127.31.235.129:32811)
0620 20:25:32.063702 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:32.063708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:32.084526 (+ 20818us) server_negotiation.cc:407] Connection header received
0620 20:25:32.085058 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:32.085064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:32.085160 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:32.085252 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:32.087309 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:32.088438 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:32.098127 (+ 9689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:32.098756 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:32.098839 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:32.103897 (+ 5058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:32.103912 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:32.103924 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:32.103969 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:32.107065 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:32.107779 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:32.107789 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:32.107798 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:32.107934 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:32.108307 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:32.108312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:32.108315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:32.108680 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:32.108994 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:32.109313 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 20:25:32.109620 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:25:32.110695 14302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:32.206235 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 14178
I20260620 20:25:32.224823 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 14313
Jun 20 20:25:32 dist-test-slave-zzwf krb5kdc[14072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.130@KRBTEST.COM
I20260620 20:25:32.243826 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:32.225679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53170 (local address 127.31.235.130:35781)
0620 20:25:32.225831 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:32.225836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:32.225857 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:25:32.226105 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:32.226109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:32.226172 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:32.226264 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:32.228109 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:32.229014 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:32.230915 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:32.231432 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:32.231546 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:32.237162 (+ 5616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:32.237178 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:32.237195 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:32.237238 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:32.240433 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:32.241416 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:32.241424 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:32.241426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:32.241688 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:32.242137 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:32.242142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:32.242144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:32.242559 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:32.242845 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:32.243241 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 20:25:32.243430 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 20:25:32.244660 14437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:32.367501 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 14313
I20260620 20:25:32.385819 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 14448
Jun 20 20:25:32 dist-test-slave-zzwf krb5kdc[14072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.131@KRBTEST.COM
I20260620 20:25:32.401222 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:32.386586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38474 (local address 127.31.235.131:42689)
0620 20:25:32.386761 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:32.386766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:32.386783 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:25:32.386916 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:32.386920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:32.386982 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:32.387068 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:32.388634 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:32.389421 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:32.390908 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:32.391318 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:32.391420 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:32.396084 (+ 4664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:32.396097 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:32.396127 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:32.396163 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:32.398726 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:32.399441 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:32.399449 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:32.399451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:32.399594 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:32.399952 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:32.399956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:32.399958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:32.400297 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:32.400510 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:32.400805 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 20:25:32.400957 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:25:32.401991 14572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:32.493532 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 14448
I20260620 20:25:32.509939 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 14085
I20260620 20:25:32.510993 14147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:32.623270 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 14085
I20260620 20:25:32.644497 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:46879
--webserver_interface=127.31.235.190
--webserver_port=36109
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:46879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:25:32.983393 14627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:32.983716 14627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:32.983876 14627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:32.991271 14627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:25:32.991389 14627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:32.991483 14627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:32.991581 14627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:25:32.991657 14627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:25:33.002317 14627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:46879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:46879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=36109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:33.003744 14627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:33.005105 14627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:33.028115 14634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:33.028146 14636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:33.028045 14633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:33.029759 14627 server_base.cc:1034] running on GCE node
I20260620 20:25:33.030678 14627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:33.032346 14627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:33.033619 14627 hybrid_clock.cc:648] HybridClock initialized: now 1781987133033564 us; error 63 us; skew 500 ppm
I20260620 20:25:33.034080 14627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:33 dist-test-slave-zzwf krb5kdc[14072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987133, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:33.037798 14627 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:25:33.039469 14627 webserver.cc:460] Webserver started at http://127.31.235.190:36109/ using document root <none> and password file <none>
I20260620 20:25:33.040223 14627 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:33.040380 14627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:33.046388 14627 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 20:25:33.049316 14643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:33.050731 14627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:25:33.050920 14627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "e7e13fdf76554383b4738bd4829178a1"
format_stamp: "Formatted at 2026-06-20 20:25:29 on dist-test-slave-zzwf"
server_key: "5e5296ec6efcf2a56479f78a7c74f25a"
server_key_iv: "cd682919ffc98391461ff3542ceb7a01"
server_key_version: "encryptionkey@0"
I20260620 20:25:33.051988 14627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:33.073839 14627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:33.078344 14627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:33.078634 14627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:33.099072 14627 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:46879
I20260620 20:25:33.099099 14694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:46879 every 8 connection(s)
I20260620 20:25:33.100672 14627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:25:33.107331 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 14627
I20260620 20:25:33.108281 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:32811
--local_ip_for_outbound_sockets=127.31.235.129
--tserver_master_addrs=127.31.235.190:46879
--webserver_port=45965
--webserver_interface=127.31.235.129
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:25:33.108594 14695 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:25:33.112644 14695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1: Bootstrap starting.
I20260620 20:25:33.142419 14695 log.cc:824] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1: Log is configured to *not* fsync() on all Append() calls
I20260620 20:25:33.158469 14695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:25:33.159170 14695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1: Bootstrap complete.
I20260620 20:25:33.173482 14695 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e13fdf76554383b4738bd4829178a1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46879 } }
I20260620 20:25:33.174961 14695 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e13fdf76554383b4738bd4829178a1, State: Initialized, Role: FOLLOWER
I20260620 20:25:33.175850 14695 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e13fdf76554383b4738bd4829178a1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46879 } }
I20260620 20:25:33.176236 14695 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:25:33.176398 14695 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:25:33.176615 14695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:25:33.183216 14695 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e13fdf76554383b4738bd4829178a1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46879 } }
I20260620 20:25:33.183905 14695 leader_election.cc:302] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7e13fdf76554383b4738bd4829178a1; no voters:
I20260620 20:25:33.184588 14695 leader_election.cc:288] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 20:25:33.184772 14699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:25:33.186167 14699 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 2 LEADER]: Becoming Leader. State: Replica: e7e13fdf76554383b4738bd4829178a1, State: Running, Role: LEADER
I20260620 20:25:33.186839 14695 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:25:33.186687 14699 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e13fdf76554383b4738bd4829178a1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46879 } }
I20260620 20:25:33.189397 14701 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7e13fdf76554383b4738bd4829178a1. Latest consensus state: current_term: 2 leader_uuid: "e7e13fdf76554383b4738bd4829178a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e13fdf76554383b4738bd4829178a1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46879 } } }
I20260620 20:25:33.189805 14701 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:25:33.190356 14700 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e7e13fdf76554383b4738bd4829178a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e13fdf76554383b4738bd4829178a1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46879 } } }
I20260620 20:25:33.190719 14700 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:25:33.201021 14706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:25:33.209380 14706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:25:33.210813 14706 catalog_manager.cc:1259] Loaded cluster ID: ce240b7860644902997330036a94842f
I20260620 20:25:33.210920 14706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:25:33.214092 14706 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 20:25:33.233597 14706 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 20:25:33.242272 14706 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:25:33.243638 14706 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1: Loaded TSK: 0
I20260620 20:25:33.244746 14706 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:25:33.500442 14697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:33.500775 14697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:33.500927 14697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:33.508419 14697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:33.508570 14697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:33.508738 14697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:25:33.518674 14697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:32811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=45965
--tserver_master_addrs=127.31.235.190:46879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:33.520033 14697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:33.521349 14697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:33.536207 14722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:33.536468 14725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:33.536952 14723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:33.537529 14697 server_base.cc:1034] running on GCE node
I20260620 20:25:33.538148 14697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:33.539196 14697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:33.540412 14697 hybrid_clock.cc:648] HybridClock initialized: now 1781987133540361 us; error 41 us; skew 500 ppm
I20260620 20:25:33.540721 14697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:33 dist-test-slave-zzwf krb5kdc[14072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987133, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:33.548956 14697 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:25:33.550711 14697 webserver.cc:460] Webserver started at http://127.31.235.129:45965/ using document root <none> and password file <none>
I20260620 20:25:33.551561 14697 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:33.551705 14697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:33.558830 14697 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:25:33.562170 14732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:33.563766 14697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:25:33.563931 14697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "cb8a664213f642c890df98ac673d6116"
format_stamp: "Formatted at 2026-06-20 20:25:30 on dist-test-slave-zzwf"
server_key: "5f330f659ddf70e14d96c17765550276"
server_key_iv: "8465c82177519e1ea9301d88b1131532"
server_key_version: "encryptionkey@0"
I20260620 20:25:33.565248 14697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:33.583309 14697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:33.588640 14697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:33.589007 14697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:33.590256 14697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:25:33.591882 14697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:25:33.591992 14697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:33.592101 14697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:25:33.592173 14697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:33.628948 14697 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:32811
I20260620 20:25:33.629037 14844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:32811 every 8 connection(s)
I20260620 20:25:33.630443 14697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:25:33.636890 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 14697
I20260620 20:25:33.637533 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:35781
--local_ip_for_outbound_sockets=127.31.235.130
--tserver_master_addrs=127.31.235.190:46879
--webserver_port=35081
--webserver_interface=127.31.235.130
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:25:33 dist-test-slave-zzwf krb5kdc[14072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987133, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:25:33.655447 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:33.633360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.129:47861)
0620 20:25:33.634188 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:33.634230 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:33.635914 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:33.637361 (+ 1447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:33.637374 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:33.638047 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:33.638579 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:33.638597 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:33.640236 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:33.640244 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:33.642495 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:33.642513 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:33.643540 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:33.643549 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:33.643875 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:33.644610 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:33.644632 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:33.646774 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:33.651249 (+ 4475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:33.651256 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:33.651268 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:33.651687 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:33.652177 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:33.652180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:33.652182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:33.652344 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:33.653301 (+ 957us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:33.653306 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:33.653649 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 20:25:33.653871 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 20:25:33.654255 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":383,"threads_started":1}
I20260620 20:25:33.656119 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:33.633270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:47861 (local address 127.31.235.190:46879)
0620 20:25:33.633913 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:33.633924 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:33.634350 (+ 426us) server_negotiation.cc:407] Connection header received
0620 20:25:33.636441 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:33.636454 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:33.636815 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:33.637217 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:33.638820 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:33.640031 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:33.642653 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:33.643309 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:33.643446 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:33.647068 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:33.647081 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:33.647102 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:33.647168 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:33.651108 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:33.651837 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:33.651849 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:33.651859 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:33.652069 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:33.652479 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:33.652487 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:33.652493 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:33.653158 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:33.653356 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:33.654153 (+ 797us) server_negotiation.cc:299] Negotiation successful
0620 20:25:33.654627 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":325,"threads_started":1}
I20260620 20:25:33.659143 14845 heartbeater.cc:347] Connected to a master server at 127.31.235.190:46879
I20260620 20:25:33.659443 14845 heartbeater.cc:464] Registering TS with master...
I20260620 20:25:33.660162 14845 heartbeater.cc:511] Master 127.31.235.190:46879 requested a full tablet report, sending...
I20260620 20:25:33.662657 14660 ts_manager.cc:194] Registered new tserver with Master: cb8a664213f642c890df98ac673d6116 (127.31.235.129:32811)
I20260620 20:25:33.664731 14660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:47861
W20260620 20:25:33.965581 14849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:33.965842 14849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:33.965934 14849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:33.973215 14849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:33.973325 14849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:33.973438 14849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:25:33.983130 14849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:35781
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=35081
--tserver_master_addrs=127.31.235.190:46879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:33.984338 14849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:33.985759 14849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:34.000003 14854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:33.999974 14855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:33.999934 14857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:34.000730 14849 server_base.cc:1034] running on GCE node
I20260620 20:25:34.001364 14849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:34.002491 14849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:34.003715 14849 hybrid_clock.cc:648] HybridClock initialized: now 1781987134003636 us; error 68 us; skew 500 ppm
I20260620 20:25:34.004052 14849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:34 dist-test-slave-zzwf krb5kdc[14072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987134, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:34.007426 14849 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:25:34.008996 14849 webserver.cc:460] Webserver started at http://127.31.235.130:35081/ using document root <none> and password file <none>
I20260620 20:25:34.009789 14849 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:34.009934 14849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:34.016047 14849 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 20:25:34.019091 14864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:34.020547 14849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:25:34.020747 14849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "2ce6988a818c4a8fb404023cf7ad39d1"
format_stamp: "Formatted at 2026-06-20 20:25:31 on dist-test-slave-zzwf"
server_key: "c08546fe3e1abc6ae55ea225d7843c2e"
server_key_iv: "e983f8375d46a1c972499f0b8098c097"
server_key_version: "encryptionkey@0"
I20260620 20:25:34.021802 14849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:34.038703 14849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:34.043275 14849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:34.043638 14849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:34.044750 14849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:25:34.046263 14849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:25:34.046409 14849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:34.046573 14849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:25:34.046659 14849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:34.081984 14849 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:35781
I20260620 20:25:34.082064 14976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:35781 every 8 connection(s)
I20260620 20:25:34.083688 14849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:25:34.092865 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 14849
I20260620 20:25:34.093544 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:42689
--local_ip_for_outbound_sockets=127.31.235.131
--tserver_master_addrs=127.31.235.190:46879
--webserver_port=32775
--webserver_interface=127.31.235.131
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:25:34 dist-test-slave-zzwf krb5kdc[14072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987134, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:25:34.105067 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:34.086456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:43709 (local address 127.31.235.190:46879)
0620 20:25:34.086626 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:34.086630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:34.087884 (+ 1254us) server_negotiation.cc:407] Connection header received
0620 20:25:34.089753 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:34.089759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:34.089841 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:34.089934 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:34.091364 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.092088 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.093868 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.094300 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.094440 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:34.097860 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:34.097865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:34.097868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:34.097913 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:34.101407 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:34.102405 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:34.102411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:34.102415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:34.102581 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:34.103089 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:34.103095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:34.103098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:34.103501 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:34.103650 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:34.104501 (+ 851us) server_negotiation.cc:299] Negotiation successful
0620 20:25:34.104732 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 20:25:34.106391 14979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:34.086942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.130:43709)
0620 20:25:34.087720 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:34.087761 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:34.089416 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:34.090134 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:34.090143 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:34.090626 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:34.091180 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:34.091197 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.092264 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.092271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:34.093636 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:34.093652 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.094520 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.094528 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:34.094786 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:34.095456 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:34.095477 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:34.097652 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:34.101568 (+ 3916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:34.101580 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:34.101598 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:34.102263 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:34.102715 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:34.102723 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:34.102729 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:34.102962 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:34.103640 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:34.103651 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:34.104090 (+ 439us) client_negotiation.cc:769] Sending connection context
0620 20:25:34.104387 (+ 297us) client_negotiation.cc:240] Negotiation successful
0620 20:25:34.104958 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":302,"threads_started":1}
I20260620 20:25:34.107898 14977 heartbeater.cc:347] Connected to a master server at 127.31.235.190:46879
I20260620 20:25:34.108237 14977 heartbeater.cc:464] Registering TS with master...
I20260620 20:25:34.109119 14977 heartbeater.cc:511] Master 127.31.235.190:46879 requested a full tablet report, sending...
I20260620 20:25:34.110775 14660 ts_manager.cc:194] Registered new tserver with Master: 2ce6988a818c4a8fb404023cf7ad39d1 (127.31.235.130:35781)
I20260620 20:25:34.112095 14660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:43709
W20260620 20:25:34.422048 14980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:34.422312 14980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:34.422403 14980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:34.430586 14980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:34.430701 14980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:34.430815 14980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:25:34.440640 14980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:42689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=32775
--tserver_master_addrs=127.31.235.190:46879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:34.441957 14980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:34.443238 14980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:34.456453 14986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:34.456473 14988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:34.456640 14985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:34.457432 14980 server_base.cc:1034] running on GCE node
I20260620 20:25:34.458164 14980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:34.459100 14980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:34.460325 14980 hybrid_clock.cc:648] HybridClock initialized: now 1781987134460255 us; error 51 us; skew 500 ppm
I20260620 20:25:34.460695 14980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:34 dist-test-slave-zzwf krb5kdc[14072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987134, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:34.464018 14980 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:25:34.465668 14980 webserver.cc:460] Webserver started at http://127.31.235.131:32775/ using document root <none> and password file <none>
I20260620 20:25:34.466370 14980 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:34.466512 14980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:34.472012 14980 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:25:34.474776 14995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:34.476100 14980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:25:34.476289 14980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "6e3716d2910a408d9793c2dd8281cebd"
format_stamp: "Formatted at 2026-06-20 20:25:31 on dist-test-slave-zzwf"
server_key: "247e4ba33aa47d599e632b51cb92b96d"
server_key_iv: "a0f021742008d264aaf3968d7f5b0057"
server_key_version: "encryptionkey@0"
I20260620 20:25:34.477254 14980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:34.501892 14980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:34.506465 14980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:34.506778 14980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:34.507927 14980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:25:34.509389 14980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:25:34.509523 14980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:34.509658 14980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:25:34.509742 14980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:34.544266 14980 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:42689
I20260620 20:25:34.544351 15107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:42689 every 8 connection(s)
I20260620 20:25:34.545842 14980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:25:34.549866 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 14980
Jun 20 20:25:34 dist-test-slave-zzwf krb5kdc[14072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987134, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:25:34.567001 15111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:34.551046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49778 (local address 127.31.235.190:46879)
0620 20:25:34.551554 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:34.551563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:34.551920 (+ 357us) server_negotiation.cc:407] Connection header received
0620 20:25:34.552067 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:34.552073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:34.552153 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:34.552267 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:34.553971 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.555048 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.556966 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.557662 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.557797 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:34.560904 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:34.560913 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:34.560920 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:34.560968 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:34.564072 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:34.564913 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:34.564917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:34.564919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:34.565043 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:34.565551 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:34.565555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:34.565557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:34.565842 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:34.566105 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:34.566402 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 20:25:34.566625 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":255,"threads_started":1}
I20260620 20:25:34.569206 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:34.548668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:50125 (local address 127.31.235.190:46879)
0620 20:25:34.548891 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:34.548898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:34.550242 (+ 1344us) server_negotiation.cc:407] Connection header received
0620 20:25:34.552845 (+ 2603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:34.552852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:34.552935 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:34.553056 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:34.554862 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.555977 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.558206 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.558844 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.558974 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:34.562806 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:34.562813 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:34.562823 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:34.562870 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:34.565886 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:34.566851 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:34.566857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:34.566860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:34.566993 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:34.567454 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:34.567459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:34.567467 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:34.567820 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:34.567964 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:34.568778 (+ 814us) server_negotiation.cc:299] Negotiation successful
0620 20:25:34.568966 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:25:34.570533 15110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:34.549156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.131:50125)
0620 20:25:34.550078 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:34.550130 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:34.552580 (+ 2450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:34.553260 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:34.553268 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:34.553842 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:34.554649 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:34.554675 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.556160 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.556164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:34.558059 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:34.558071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.559075 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.559080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:34.559301 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:34.560052 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:34.560079 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:34.562570 (+ 2491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:34.566051 (+ 3481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:34.566059 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:34.566076 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:34.566717 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:34.567107 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:34.567114 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:34.567120 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:34.567332 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:34.567956 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:34.567962 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:34.568396 (+ 434us) client_negotiation.cc:769] Sending connection context
0620 20:25:34.568684 (+ 288us) client_negotiation.cc:240] Negotiation successful
0620 20:25:34.569265 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":419,"threads_started":1}
I20260620 20:25:34.572005 15108 heartbeater.cc:347] Connected to a master server at 127.31.235.190:46879
I20260620 20:25:34.572328 15108 heartbeater.cc:464] Registering TS with master...
I20260620 20:25:34.573053 15108 heartbeater.cc:511] Master 127.31.235.190:46879 requested a full tablet report, sending...
I20260620 20:25:34.574504 14660 ts_manager.cc:194] Registered new tserver with Master: 6e3716d2910a408d9793c2dd8281cebd (127.31.235.131:42689)
I20260620 20:25:34.575317 14660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:50125
I20260620 20:25:34.582253 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:25:34.588902 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:34.583042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49784 (local address 127.31.235.190:46879)
0620 20:25:34.583267 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:34.583277 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:34.583719 (+ 442us) server_negotiation.cc:407] Connection header received
0620 20:25:34.583877 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:34.583880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:34.584008 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:34.584128 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:25:34.584501 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.585261 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.586729 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.587153 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.587247 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:34.587804 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:25:34.588320 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:25:34.588436 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:34.588554 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 20:25:34.588620 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:25:34.589911 14660 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:25:34.597162 14660 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:25:34.631022 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:34.609954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:32811 (local address 127.0.0.1:44252)
0620 20:25:34.620461 (+ 10507us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:34.620497 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:34.620738 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:34.621538 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:34.621546 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:34.621585 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:34.622004 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:34.622017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.624692 (+ 2675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.624701 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:34.627762 (+ 3061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:34.627778 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.630018 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.630024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:34.630226 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:34.630280 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 20:25:34.630414 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 20:25:34.630637 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10291,"thread_start_us":10179,"threads_started":1}
I20260620 20:25:34.632041 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:34.610174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44252 (local address 127.31.235.129:32811)
0620 20:25:34.620012 (+ 9838us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:34.620023 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:34.620643 (+ 620us) server_negotiation.cc:407] Connection header received
0620 20:25:34.620965 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:34.620975 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:34.621275 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:34.622219 (+ 944us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:34.622449 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.624504 (+ 2055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.627953 (+ 3449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.629442 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.629653 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:34.629843 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:34.631456 (+ 1613us) server_negotiation.cc:299] Negotiation successful
0620 20:25:34.631679 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9666,"thread_start_us":9552,"threads_started":1}
I20260620 20:25:34.633719 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:34.610229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:35781 (local address 127.0.0.1:53186)
0620 20:25:34.621395 (+ 11166us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:34.621414 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:34.621551 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:34.622228 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:34.622236 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:34.622275 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:34.622755 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:34.622770 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.627104 (+ 4334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.627110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:34.630391 (+ 3281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:34.630405 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.632931 (+ 2526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.632937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:34.633107 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:34.633139 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 20:25:34.633227 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 20:25:34.633360 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11032,"thread_start_us":10886,"threads_started":1}
I20260620 20:25:34.635401 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:34.610492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.31.235.130:35781)
0620 20:25:34.619646 (+ 9154us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:34.619660 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:34.621674 (+ 2014us) server_negotiation.cc:407] Connection header received
0620 20:25:34.621787 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:34.621794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:34.622017 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:34.622221 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:34.623014 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.626865 (+ 3851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.630965 (+ 4100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.632435 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.632599 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:34.632770 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:34.634742 (+ 1972us) server_negotiation.cc:299] Negotiation successful
0620 20:25:34.634941 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8910,"thread_start_us":8759,"threads_started":1}
I20260620 20:25:34.636359 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:34.621895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38488 (local address 127.31.235.131:42689)
0620 20:25:34.626668 (+ 4773us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:34.626677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:34.627404 (+ 727us) server_negotiation.cc:407] Connection header received
0620 20:25:34.627590 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:34.627598 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:34.627847 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:34.628106 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:34.628730 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.630397 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.632821 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.634408 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.634764 (+ 356us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:34.634941 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:34.635718 (+ 777us) server_negotiation.cc:299] Negotiation successful
0620 20:25:34.635911 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4593,"thread_start_us":4407,"threads_started":1}
I20260620 20:25:34.637181 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:34.622572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:42689 (local address 127.0.0.1:38488)
0620 20:25:34.627269 (+ 4697us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:34.627288 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:34.627427 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:34.628086 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:34.628090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:34.628117 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:34.628524 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:34.628540 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.630572 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.630576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:34.632646 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:34.632660 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.635127 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.635135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:34.635359 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:34.635408 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 20:25:34.636593 (+ 1185us) client_negotiation.cc:240] Negotiation successful
0620 20:25:34.636772 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4529,"thread_start_us":4353,"threads_started":1}
I20260620 20:25:34.636900 14780 tablet_service.cc:1467] Processing CreateTablet for tablet fe20ffe79c6647a9b399f71091259388 (DEFAULT_TABLE table=test-table [id=6dd22288cb434742a2079ae11aae8504]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:25:34.638181 14780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe20ffe79c6647a9b399f71091259388. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:25:34.639369 14912 tablet_service.cc:1467] Processing CreateTablet for tablet fe20ffe79c6647a9b399f71091259388 (DEFAULT_TABLE table=test-table [id=6dd22288cb434742a2079ae11aae8504]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:25:34.640652 14912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe20ffe79c6647a9b399f71091259388. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:25:34.640926 15043 tablet_service.cc:1467] Processing CreateTablet for tablet fe20ffe79c6647a9b399f71091259388 (DEFAULT_TABLE table=test-table [id=6dd22288cb434742a2079ae11aae8504]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:25:34.642191 15043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe20ffe79c6647a9b399f71091259388. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:25:34.653744 15124 tablet_bootstrap.cc:492] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1: Bootstrap starting.
I20260620 20:25:34.657449 15124 tablet_bootstrap.cc:654] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1: Neither blocks nor log segments found. Creating new log.
I20260620 20:25:34.658710 15124 log.cc:824] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1: Log is configured to *not* fsync() on all Append() calls
I20260620 20:25:34.659277 15126 tablet_bootstrap.cc:492] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd: Bootstrap starting.
I20260620 20:25:34.664177 15125 tablet_bootstrap.cc:492] T fe20ffe79c6647a9b399f71091259388 P cb8a664213f642c890df98ac673d6116: Bootstrap starting.
I20260620 20:25:34.667925 15124 tablet_bootstrap.cc:492] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1: No bootstrap required, opened a new log
I20260620 20:25:34.668087 15125 tablet_bootstrap.cc:654] T fe20ffe79c6647a9b399f71091259388 P cb8a664213f642c890df98ac673d6116: Neither blocks nor log segments found. Creating new log.
I20260620 20:25:34.668088 15126 tablet_bootstrap.cc:654] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd: Neither blocks nor log segments found. Creating new log.
I20260620 20:25:34.668200 15124 ts_tablet_manager.cc:1397] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260620 20:25:34.669350 15125 log.cc:824] T fe20ffe79c6647a9b399f71091259388 P cb8a664213f642c890df98ac673d6116: Log is configured to *not* fsync() on all Append() calls
I20260620 20:25:34.669442 15126 log.cc:824] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd: Log is configured to *not* fsync() on all Append() calls
I20260620 20:25:34.671051 14845 heartbeater.cc:503] Master 127.31.235.190:46879 was elected leader, sending a full tablet report...
I20260620 20:25:34.679672 15126 tablet_bootstrap.cc:492] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd: No bootstrap required, opened a new log
I20260620 20:25:34.679757 15125 tablet_bootstrap.cc:492] T fe20ffe79c6647a9b399f71091259388 P cb8a664213f642c890df98ac673d6116: No bootstrap required, opened a new log
I20260620 20:25:34.680003 15126 ts_tablet_manager.cc:1397] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.004s
I20260620 20:25:34.680030 15125 ts_tablet_manager.cc:1397] T fe20ffe79c6647a9b399f71091259388 P cb8a664213f642c890df98ac673d6116: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260620 20:25:34.680989 15124 raft_consensus.cc:348] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 } } peers { permanent_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35781 } } peers { permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 } }
I20260620 20:25:34.681476 15124 raft_consensus.cc:374] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:25:34.681610 15124 raft_consensus.cc:729] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce6988a818c4a8fb404023cf7ad39d1, State: Initialized, Role: FOLLOWER
I20260620 20:25:34.682214 15124 consensus_queue.cc:260] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 } } peers { permanent_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35781 } } peers { permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 } }
I20260620 20:25:34.683423 14977 heartbeater.cc:503] Master 127.31.235.190:46879 was elected leader, sending a full tablet report...
I20260620 20:25:34.683996 15124 ts_tablet_manager.cc:1428] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260620 20:25:34.689881 15125 raft_consensus.cc:348] T fe20ffe79c6647a9b399f71091259388 P cb8a664213f642c890df98ac673d6116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 } } peers { permanent_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35781 } } peers { permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 } }
I20260620 20:25:34.690363 15125 raft_consensus.cc:374] T fe20ffe79c6647a9b399f71091259388 P cb8a664213f642c890df98ac673d6116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:25:34.690493 15125 raft_consensus.cc:729] T fe20ffe79c6647a9b399f71091259388 P cb8a664213f642c890df98ac673d6116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8a664213f642c890df98ac673d6116, State: Initialized, Role: FOLLOWER
I20260620 20:25:34.690652 15126 raft_consensus.cc:348] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 } } peers { permanent_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35781 } } peers { permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 } }
I20260620 20:25:34.691030 15125 consensus_queue.cc:260] T fe20ffe79c6647a9b399f71091259388 P cb8a664213f642c890df98ac673d6116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 } } peers { permanent_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35781 } } peers { permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 } }
I20260620 20:25:34.691205 15126 raft_consensus.cc:374] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:25:34.691329 15126 raft_consensus.cc:729] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e3716d2910a408d9793c2dd8281cebd, State: Initialized, Role: FOLLOWER
I20260620 20:25:34.692000 15126 consensus_queue.cc:260] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 } } peers { permanent_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35781 } } peers { permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 } }
I20260620 20:25:34.692787 15125 ts_tablet_manager.cc:1428] T fe20ffe79c6647a9b399f71091259388 P cb8a664213f642c890df98ac673d6116: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 20:25:34.693169 15108 heartbeater.cc:503] Master 127.31.235.190:46879 was elected leader, sending a full tablet report...
I20260620 20:25:34.693885 15126 ts_tablet_manager.cc:1428] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260620 20:25:34.760862 15130 raft_consensus.cc:482] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:25:34.761166 15130 raft_consensus.cc:504] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 } } peers { permanent_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35781 } } peers { permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 } }
I20260620 20:25:34.762862 15130 leader_election.cc:288] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb8a664213f642c890df98ac673d6116 (127.31.235.129:32811), 6e3716d2910a408d9793c2dd8281cebd (127.31.235.131:42689)
I20260620 20:25:34.772382 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:34.763496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:35471 (local address 127.31.235.131:42689)
0620 20:25:34.763745 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:34.763754 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:34.764256 (+ 502us) server_negotiation.cc:407] Connection header received
0620 20:25:34.764534 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:34.764539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:34.764611 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:34.764709 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:34.765543 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.766680 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.769675 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.771174 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.771331 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:34.771399 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:34.771926 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 20:25:34.772127 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:25:34.772365 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:34.763277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:42759 (local address 127.31.235.129:32811)
0620 20:25:34.763523 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:34.763532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:34.764430 (+ 898us) server_negotiation.cc:407] Connection header received
0620 20:25:34.764556 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:34.764561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:34.764632 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:34.764728 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:34.765602 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.766683 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.769675 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.771117 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.771316 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:34.771398 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:34.771926 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 20:25:34.772071 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:25:34.772552 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:34.763218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:32811 (local address 127.31.235.130:42759)
0620 20:25:34.764323 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:34.764337 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:34.764440 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:34.764787 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:34.764793 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:34.764832 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:34.765431 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:34.765444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.766818 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.766821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:34.769468 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:34.769503 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.771408 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.771415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:34.771730 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:34.771778 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 20:25:34.771870 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 20:25:34.772060 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"mutex_wait_us":291,"spinlock_wait_cycles":896,"thread_start_us":542,"threads_started":1}
I20260620 20:25:34.772701 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:34.763197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:42689 (local address 127.31.235.130:35471)
0620 20:25:34.764146 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:34.764161 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:34.764334 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:34.764787 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:34.764793 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:34.764838 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:34.765343 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:34.765361 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.766816 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.766821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:34.769468 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:34.769483 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.771414 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.771419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:34.771714 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:34.771771 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 20:25:34.771870 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 20:25:34.772060 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":265,"spinlock_wait_cycles":5760,"thread_start_us":794,"threads_started":1}
I20260620 20:25:34.773806 15063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe20ffe79c6647a9b399f71091259388" candidate_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3716d2910a408d9793c2dd8281cebd" is_pre_election: true
I20260620 20:25:34.773808 14800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe20ffe79c6647a9b399f71091259388" candidate_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb8a664213f642c890df98ac673d6116" is_pre_election: true
I20260620 20:25:34.774392 14800 raft_consensus.cc:2454] T fe20ffe79c6647a9b399f71091259388 P cb8a664213f642c890df98ac673d6116 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ce6988a818c4a8fb404023cf7ad39d1 in term 0.
I20260620 20:25:34.774389 15063 raft_consensus.cc:2454] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ce6988a818c4a8fb404023cf7ad39d1 in term 0.
I20260620 20:25:34.775087 14865 leader_election.cc:302] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ce6988a818c4a8fb404023cf7ad39d1, 6e3716d2910a408d9793c2dd8281cebd; no voters:
I20260620 20:25:34.775454 15130 raft_consensus.cc:2781] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:25:34.775597 15130 raft_consensus.cc:482] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:25:34.775725 15130 raft_consensus.cc:3037] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:25:34.778103 15130 raft_consensus.cc:504] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 } } peers { permanent_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35781 } } peers { permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 } }
I20260620 20:25:34.778782 15130 leader_election.cc:288] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 [CANDIDATE]: Term 1 election: Requested vote from peers cb8a664213f642c890df98ac673d6116 (127.31.235.129:32811), 6e3716d2910a408d9793c2dd8281cebd (127.31.235.131:42689)
I20260620 20:25:34.779198 14800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe20ffe79c6647a9b399f71091259388" candidate_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb8a664213f642c890df98ac673d6116"
I20260620 20:25:34.779294 15063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe20ffe79c6647a9b399f71091259388" candidate_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e3716d2910a408d9793c2dd8281cebd"
I20260620 20:25:34.779441 14800 raft_consensus.cc:3037] T fe20ffe79c6647a9b399f71091259388 P cb8a664213f642c890df98ac673d6116 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:25:34.779512 15063 raft_consensus.cc:3037] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:25:34.782058 14800 raft_consensus.cc:2454] T fe20ffe79c6647a9b399f71091259388 P cb8a664213f642c890df98ac673d6116 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ce6988a818c4a8fb404023cf7ad39d1 in term 1.
I20260620 20:25:34.782061 15063 raft_consensus.cc:2454] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ce6988a818c4a8fb404023cf7ad39d1 in term 1.
I20260620 20:25:34.782473 14866 leader_election.cc:302] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ce6988a818c4a8fb404023cf7ad39d1, cb8a664213f642c890df98ac673d6116; no voters:
I20260620 20:25:34.782791 15130 raft_consensus.cc:2781] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:25:34.783223 15130 raft_consensus.cc:686] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 1 LEADER]: Becoming Leader. State: Replica: 2ce6988a818c4a8fb404023cf7ad39d1, State: Running, Role: LEADER
I20260620 20:25:34.783679 15130 consensus_queue.cc:237] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 } } peers { permanent_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35781 } } peers { permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 } }
I20260620 20:25:34.788424 14659 catalog_manager.cc:5495] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ce6988a818c4a8fb404023cf7ad39d1 (127.31.235.130). New cstate: current_term: 1 leader_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ce6988a818c4a8fb404023cf7ad39d1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 35781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 } health_report { overall_health: UNKNOWN } } }
W20260620 20:25:34.797438 15109 tablet.cc:2369] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:25:34.828775 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:34.823493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.31.235.130:35781)
0620 20:25:34.823705 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:34.823714 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:34.823734 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:25:34.823834 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:34.823837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:34.823919 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:34.824072 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:25:34.824434 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.825201 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.826639 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.827043 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.827188 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:34.827691 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:25:34.828170 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:25:34.828321 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:34.828416 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 20:25:34.828492 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 20:25:34.835884 14978 tablet.cc:2369] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:25:34.836769 15063 raft_consensus.cc:1261] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd [term 1 FOLLOWER]: Refusing update from remote peer 2ce6988a818c4a8fb404023cf7ad39d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:25:34.836769 14800 raft_consensus.cc:1261] T fe20ffe79c6647a9b399f71091259388 P cb8a664213f642c890df98ac673d6116 [term 1 FOLLOWER]: Refusing update from remote peer 2ce6988a818c4a8fb404023cf7ad39d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:25:34.837813 15130 consensus_queue.cc:1035] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb8a664213f642c890df98ac673d6116" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 32811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:25:34.838105 15135 consensus_queue.cc:1035] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e3716d2910a408d9793c2dd8281cebd" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:25:34.871549 14659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:25:34.871825 14659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:25:34.875422 14659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1: Sending DeleteTablet for 3 replicas of tablet fe20ffe79c6647a9b399f71091259388
I20260620 20:25:34.876524 14780 tablet_service.cc:1514] Processing DeleteTablet for tablet fe20ffe79c6647a9b399f71091259388 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:25:34 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:44252
I20260620 20:25:34.876796 14912 tablet_service.cc:1514] Processing DeleteTablet for tablet fe20ffe79c6647a9b399f71091259388 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:25:34 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:53186
I20260620 20:25:34.877039 15043 tablet_service.cc:1514] Processing DeleteTablet for tablet fe20ffe79c6647a9b399f71091259388 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:25:34 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:38488
I20260620 20:25:34.877942 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 14697
I20260620 20:25:34.893291 15147 tablet_replica.cc:330] T fe20ffe79c6647a9b399f71091259388 P cb8a664213f642c890df98ac673d6116: stopping tablet replica
I20260620 20:25:34.894065 15147 raft_consensus.cc:2229] T fe20ffe79c6647a9b399f71091259388 P cb8a664213f642c890df98ac673d6116 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:25:34.894807 15147 raft_consensus.cc:2258] T fe20ffe79c6647a9b399f71091259388 P cb8a664213f642c890df98ac673d6116 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:25:34.897269 15147 ts_tablet_manager.cc:1905] T fe20ffe79c6647a9b399f71091259388 P cb8a664213f642c890df98ac673d6116: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:25:34.906476 15147 ts_tablet_manager.cc:1918] T fe20ffe79c6647a9b399f71091259388 P cb8a664213f642c890df98ac673d6116: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:25:34.906751 15147 log.cc:1196] T fe20ffe79c6647a9b399f71091259388 P cb8a664213f642c890df98ac673d6116: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/fe20ffe79c6647a9b399f71091259388
I20260620 20:25:34.907503 15147 ts_tablet_manager.cc:1939] T fe20ffe79c6647a9b399f71091259388 P cb8a664213f642c890df98ac673d6116: Deleting consensus metadata
I20260620 20:25:34.909739 14645 catalog_manager.cc:4841] TS cb8a664213f642c890df98ac673d6116 (127.31.235.129:32811): tablet fe20ffe79c6647a9b399f71091259388 (table test-table [id=6dd22288cb434742a2079ae11aae8504]) successfully deleted
I20260620 20:25:34.910317 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:34.879166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44258 (local address 127.31.235.129:32811)
0620 20:25:34.879386 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:34.879394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:34.880030 (+ 636us) server_negotiation.cc:407] Connection header received
0620 20:25:34.880107 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:34.880116 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:34.880214 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:34.880321 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:34.882907 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.884155 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.889401 (+ 5246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:34.890506 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:34.890654 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:34.899707 (+ 9053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:34.899729 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:34.899749 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:34.899806 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:34.904476 (+ 4670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:34.905549 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:34.905559 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:34.905563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:34.905728 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:34.907089 (+ 1361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:34.907095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:34.907099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:34.907685 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:34.907982 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:34.909656 (+ 1674us) server_negotiation.cc:299] Negotiation successful
0620 20:25:34.909915 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:25:34.911114 14840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:34.914541 15149 tablet_replica.cc:330] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd: stopping tablet replica
I20260620 20:25:34.915179 15149 raft_consensus.cc:2229] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:25:34.915848 15149 raft_consensus.cc:2258] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:25:34.916992 15148 tablet_replica.cc:330] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1: stopping tablet replica
I20260620 20:25:34.917599 15148 raft_consensus.cc:2229] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:25:34.918468 15148 raft_consensus.cc:2258] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:25:34.919984 15149 ts_tablet_manager.cc:1905] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:25:34.920789 15148 ts_tablet_manager.cc:1905] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:25:34.927728 15149 ts_tablet_manager.cc:1918] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:25:34.927949 15149 log.cc:1196] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/fe20ffe79c6647a9b399f71091259388
I20260620 20:25:34.928547 15149 ts_tablet_manager.cc:1939] T fe20ffe79c6647a9b399f71091259388 P 6e3716d2910a408d9793c2dd8281cebd: Deleting consensus metadata
I20260620 20:25:34.929414 15148 ts_tablet_manager.cc:1918] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:25:34.929629 15148 log.cc:1196] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/fe20ffe79c6647a9b399f71091259388
I20260620 20:25:34.930364 15148 ts_tablet_manager.cc:1939] T fe20ffe79c6647a9b399f71091259388 P 2ce6988a818c4a8fb404023cf7ad39d1: Deleting consensus metadata
I20260620 20:25:34.930639 14644 catalog_manager.cc:4841] TS 6e3716d2910a408d9793c2dd8281cebd (127.31.235.131:42689): tablet fe20ffe79c6647a9b399f71091259388 (table test-table [id=6dd22288cb434742a2079ae11aae8504]) successfully deleted
I20260620 20:25:34.932156 14644 catalog_manager.cc:4841] TS 2ce6988a818c4a8fb404023cf7ad39d1 (127.31.235.130:35781): tablet fe20ffe79c6647a9b399f71091259388 (table test-table [id=6dd22288cb434742a2079ae11aae8504]) successfully deleted
I20260620 20:25:35.014796 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 14697
I20260620 20:25:35.038483 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 14849
I20260620 20:25:35.054855 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:35.039300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.31.235.130:35781)
0620 20:25:35.039476 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:35.039481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:35.039594 (+ 113us) server_negotiation.cc:407] Connection header received
0620 20:25:35.039771 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:35.039775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:35.039850 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:35.039942 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:35.041822 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:35.042605 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:35.044510 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:35.044932 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:35.045019 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:35.048586 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:35.048613 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:35.048631 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:35.048668 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:35.051947 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:35.052814 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:35.052821 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:35.052824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:35.052975 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:35.053365 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:35.053368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:35.053371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:35.053794 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:35.054027 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:35.054350 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 20:25:35.054515 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:25:35.055661 14972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:35.179538 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 14849
I20260620 20:25:35.199935 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 14980
I20260620 20:25:35.213753 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:35.200765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38500 (local address 127.31.235.131:42689)
0620 20:25:35.200966 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:35.200971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:35.200990 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:25:35.201246 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:35.201251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:35.201334 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:35.201435 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:35.203130 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:35.203847 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:35.205166 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:35.205749 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:35.205855 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:35.208409 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:35.208433 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:35.208450 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:35.208499 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:35.211329 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:35.211990 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:35.211997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:35.212007 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:35.212153 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:35.212479 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:35.212482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:35.212484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:35.212824 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:35.213045 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:35.213246 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 20:25:35.213403 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 20:25:35.214390 15103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:35.302783 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 14980
I20260620 20:25:35.319456 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 14627
I20260620 20:25:35.320598 14690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:35.434285 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 14627
I20260620 20:25:35.455992 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:46879
--webserver_interface=127.31.235.190
--webserver_port=36109
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:46879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:25:35.778810 15154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:35.779094 15154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:35.779189 15154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:35.786484 15154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:25:35.786593 15154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:35.786639 15154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:35.786689 15154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:25:35.786732 15154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:25:35.796500 15154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:46879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:46879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=36109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:35.797791 15154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:35.799014 15154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:35.810782 15159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:35.810806 15162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:35.810734 15160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:35.811678 15154 server_base.cc:1034] running on GCE node
I20260620 20:25:35.812330 15154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:35.813627 15154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:35.814821 15154 hybrid_clock.cc:648] HybridClock initialized: now 1781987135814767 us; error 49 us; skew 500 ppm
I20260620 20:25:35.815099 15154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:35 dist-test-slave-zzwf krb5kdc[14072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987135, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:35.818164 15154 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:25:35.819512 15154 webserver.cc:460] Webserver started at http://127.31.235.190:36109/ using document root <none> and password file <none>
I20260620 20:25:35.820130 15154 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:35.820227 15154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:35.825317 15154 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260620 20:25:35.828023 15169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:35.829275 15154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:25:35.829407 15154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "e7e13fdf76554383b4738bd4829178a1"
format_stamp: "Formatted at 2026-06-20 20:25:29 on dist-test-slave-zzwf"
server_key: "5e5296ec6efcf2a56479f78a7c74f25a"
server_key_iv: "cd682919ffc98391461ff3542ceb7a01"
server_key_version: "encryptionkey@0"
I20260620 20:25:35.830374 15154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:35.858275 15154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:35.863097 15154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:35.863349 15154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:35.883033 15154 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:46879
I20260620 20:25:35.883052 15220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:46879 every 8 connection(s)
I20260620 20:25:35.884616 15154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:25:35.887913 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 15154
I20260620 20:25:35.888770 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:32811
--local_ip_for_outbound_sockets=127.31.235.129
--tserver_master_addrs=127.31.235.190:46879
--webserver_port=45965
--webserver_interface=127.31.235.129
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:25:35.892087 15221 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:25:35.895520 15221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1: Bootstrap starting.
I20260620 20:25:35.926832 15221 log.cc:824] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1: Log is configured to *not* fsync() on all Append() calls
I20260620 20:25:35.948378 15221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:25:35.949150 15221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1: Bootstrap complete.
I20260620 20:25:35.964107 15221 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e13fdf76554383b4738bd4829178a1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46879 } }
I20260620 20:25:35.965332 15221 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e13fdf76554383b4738bd4829178a1, State: Initialized, Role: FOLLOWER
I20260620 20:25:35.966089 15221 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e13fdf76554383b4738bd4829178a1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46879 } }
I20260620 20:25:35.966369 15221 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:25:35.966492 15221 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:25:35.966681 15221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 20:25:35.972224 15221 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e13fdf76554383b4738bd4829178a1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46879 } }
I20260620 20:25:35.972820 15221 leader_election.cc:302] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7e13fdf76554383b4738bd4829178a1; no voters:
I20260620 20:25:35.973513 15221 leader_election.cc:288] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 20:25:35.973742 15225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 20:25:35.975265 15225 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [term 3 LEADER]: Becoming Leader. State: Replica: e7e13fdf76554383b4738bd4829178a1, State: Running, Role: LEADER
I20260620 20:25:35.975759 15221 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:25:35.975876 15225 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e13fdf76554383b4738bd4829178a1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46879 } }
I20260620 20:25:35.978477 15227 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7e13fdf76554383b4738bd4829178a1. Latest consensus state: current_term: 3 leader_uuid: "e7e13fdf76554383b4738bd4829178a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e13fdf76554383b4738bd4829178a1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46879 } } }
I20260620 20:25:35.978892 15227 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:25:35.979460 15226 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e7e13fdf76554383b4738bd4829178a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e13fdf76554383b4738bd4829178a1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46879 } } }
I20260620 20:25:35.979769 15226 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e13fdf76554383b4738bd4829178a1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:25:35.988849 15232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:25:35.998586 15232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:25:36.000273 15232 catalog_manager.cc:1259] Loaded cluster ID: ce240b7860644902997330036a94842f
I20260620 20:25:36.000396 15232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 20:25:36.002761 15232 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff2d8ac59ad google::LogMessage::Fail() at ??:0
@ 0x7ff2d8ac888a google::LogMessage::SendToLog() at ??:0
@ 0x7ff2d8ac54de google::LogMessage::Flush() at ??:0
@ 0x7ff2d8ac6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff2ee4fc512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff2ee4f9976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff2ee581610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff2ee581439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff2db711345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff2db71491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20260620 20:25:36.045759 15246 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7ff2db714799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff2db6efeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff2de9c96db start_thread at ??:0
@ 0x7ff2d795f61f clone at ??:0
W20260620 20:25:36.326211 15223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:36.326570 15223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:36.326707 15223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:36.333910 15223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:36.334015 15223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:36.334126 15223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:25:36.343863 15223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:32811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=45965
--tserver_master_addrs=127.31.235.190:46879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:36.345098 15223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:36.346405 15223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:36.359589 15262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:36.359537 15263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:36.359629 15265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:36.360666 15223 server_base.cc:1034] running on GCE node
I20260620 20:25:36.361281 15223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:36.362290 15223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:36.363517 15223 hybrid_clock.cc:648] HybridClock initialized: now 1781987136363473 us; error 58 us; skew 500 ppm
I20260620 20:25:36.363837 15223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:36 dist-test-slave-zzwf krb5kdc[14072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987136, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:36.367023 15223 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:25:36.368520 15223 webserver.cc:460] Webserver started at http://127.31.235.129:45965/ using document root <none> and password file <none>
I20260620 20:25:36.369194 15223 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:36.369340 15223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:36.374663 15223 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:25:36.377408 15272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:36.378827 15223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:25:36.379011 15223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "cb8a664213f642c890df98ac673d6116"
format_stamp: "Formatted at 2026-06-20 20:25:30 on dist-test-slave-zzwf"
server_key: "5f330f659ddf70e14d96c17765550276"
server_key_iv: "8465c82177519e1ea9301d88b1131532"
server_key_version: "encryptionkey@0"
I20260620 20:25:36.380003 15223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:36.393057 15223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:36.397167 15223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:36.397471 15223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:36.398523 15223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:25:36.399870 15223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:25:36.400009 15223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:36.400137 15223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:25:36.400228 15223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:36.435411 15223 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:32811
I20260620 20:25:36.435499 15384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:32811 every 8 connection(s)
I20260620 20:25:36.436877 15223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:25:36.441706 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:36.439792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.129:33497)
0620 20:25:36.440518 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:36.440714 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":306,"threads_started":1}
W20260620 20:25:36.442371 15385 heartbeater.cc:644] Failed to heartbeat to 127.31.235.190:46879 (0 consecutive failures): Network error: Failed to ping master at 127.31.235.190:46879: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
I20260620 20:25:36.443125 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:36.442813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.129:38249)
0620 20:25:36.442967 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:36.443028 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 20:25:36.443920 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:36.443604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.129:48887)
0620 20:25:36.443739 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:36.443815 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 20:25:36.444144 15385 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 20:25:36.446839 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 15223
I20260620 20:25:36.447443 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:35781
--local_ip_for_outbound_sockets=127.31.235.130
--tserver_master_addrs=127.31.235.190:46879
--webserver_port=35081
--webserver_interface=127.31.235.130
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:25:36.778882 15388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:36.779223 15388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:36.779402 15388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:36.787040 15388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:36.787199 15388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:36.787371 15388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:25:36.797161 15388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:35781
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=35081
--tserver_master_addrs=127.31.235.190:46879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:36.798578 15388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:36.799903 15388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:36.813769 15393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:36.813747 15394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:36.814230 15396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:36.814729 15388 server_base.cc:1034] running on GCE node
I20260620 20:25:36.815342 15388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:36.816298 15388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:36.817605 15388 hybrid_clock.cc:648] HybridClock initialized: now 1781987136817532 us; error 67 us; skew 500 ppm
I20260620 20:25:36.817930 15388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:36 dist-test-slave-zzwf krb5kdc[14072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987136, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:36.821012 15388 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:25:36.822522 15388 webserver.cc:460] Webserver started at http://127.31.235.130:35081/ using document root <none> and password file <none>
I20260620 20:25:36.823233 15388 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:36.823383 15388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:36.828820 15388 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 20:25:36.831666 15403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:36.832948 15388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:25:36.833134 15388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "2ce6988a818c4a8fb404023cf7ad39d1"
format_stamp: "Formatted at 2026-06-20 20:25:31 on dist-test-slave-zzwf"
server_key: "c08546fe3e1abc6ae55ea225d7843c2e"
server_key_iv: "e983f8375d46a1c972499f0b8098c097"
server_key_version: "encryptionkey@0"
I20260620 20:25:36.834223 15388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:36.882632 15388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:36.887315 15388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:36.887617 15388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:36.888675 15388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:25:36.890197 15388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:25:36.890295 15388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:36.890441 15388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:25:36.890542 15388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:36.923566 15388 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:35781
I20260620 20:25:36.923656 15515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:35781 every 8 connection(s)
I20260620 20:25:36.925026 15388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:25:36.929960 15518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:36.928077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.130:55155)
0620 20:25:36.928782 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:36.929005 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":273,"threads_started":1}
W20260620 20:25:36.930709 15516 heartbeater.cc:644] Failed to heartbeat to 127.31.235.190:46879 (0 consecutive failures): Network error: Failed to ping master at 127.31.235.190:46879: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
I20260620 20:25:36.931119 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 15388
I20260620 20:25:36.931763 15518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:36.931338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.130:60607)
0620 20:25:36.931511 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:36.931607 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 20:25:36.931900 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:42689
--local_ip_for_outbound_sockets=127.31.235.131
--tserver_master_addrs=127.31.235.190:46879
--webserver_port=32775
--webserver_interface=127.31.235.131
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:25:36.932828 15518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:36.932365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.130:51397)
0620 20:25:36.932550 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:36.932664 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 20:25:36.933086 15516 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 20:25:37.259689 15519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:37.259974 15519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:37.260066 15519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:37.267457 15519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:37.267571 15519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:37.267683 15519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:25:37.277545 15519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:42689
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=32775
--tserver_master_addrs=127.31.235.190:46879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:37.278790 15519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:37.280069 15519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:37.294195 15525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:37.294823 15519 server_base.cc:1034] running on GCE node
W20260620 20:25:37.294943 15527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:37.295122 15524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:37.295848 15519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:37.296810 15519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:37.298036 15519 hybrid_clock.cc:648] HybridClock initialized: now 1781987137297990 us; error 37 us; skew 500 ppm
I20260620 20:25:37.298367 15519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:37 dist-test-slave-zzwf krb5kdc[14072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987137, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:37.301610 15519 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:25:37.303097 15519 webserver.cc:460] Webserver started at http://127.31.235.131:32775/ using document root <none> and password file <none>
I20260620 20:25:37.303800 15519 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:37.303947 15519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:37.309471 15519 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:25:37.312227 15534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:37.313656 15519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 20:25:37.313803 15519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "6e3716d2910a408d9793c2dd8281cebd"
format_stamp: "Formatted at 2026-06-20 20:25:31 on dist-test-slave-zzwf"
server_key: "247e4ba33aa47d599e632b51cb92b96d"
server_key_iv: "a0f021742008d264aaf3968d7f5b0057"
server_key_version: "encryptionkey@0"
I20260620 20:25:37.314786 15519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:37.345714 15519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:37.350077 15519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:37.350317 15519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:37.351305 15519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:25:37.352618 15519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:25:37.352715 15519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:37.352797 15519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:25:37.352854 15519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:37.386639 15519 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:42689
I20260620 20:25:37.386735 15646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:42689 every 8 connection(s)
I20260620 20:25:37.388098 15519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:25:37.392818 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:37.391099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.131:38811)
0620 20:25:37.391778 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:37.391983 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":269,"threads_started":1}
W20260620 20:25:37.393590 15647 heartbeater.cc:644] Failed to heartbeat to 127.31.235.190:46879 (0 consecutive failures): Network error: Failed to ping master at 127.31.235.190:46879: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
I20260620 20:25:37.394471 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:37.394114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.131:54251)
0620 20:25:37.394289 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:37.394366 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 20:25:37.395402 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:37.395114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.131:40551)
0620 20:25:37.395234 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:37.395303 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 20:25:37.395723 15647 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 20:25:37.396100 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 15519
I20260620 20:25:37.445652 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:37.444748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.129:43209)
0620 20:25:37.445209 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:37.445336 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":211,"threads_started":1}
I20260620 20:25:37.934834 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:37.933853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.130:52463)
0620 20:25:37.934366 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:37.934484 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":231,"threads_started":1}
I20260620 20:25:38.397099 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:38.396241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.131:48141)
0620 20:25:38.396746 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:38.396865 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":261,"threads_started":1}
I20260620 20:25:38.447307 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:38.446541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.129:45409)
0620 20:25:38.447025 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:38.447117 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":245,"threads_started":1}
I20260620 20:25:38.936517 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:38.935820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.130:43863)
0620 20:25:38.936264 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:38.936349 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":211,"threads_started":1}
I20260620 20:25:39.398911 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:39.398161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.131:39763)
0620 20:25:39.398653 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:39.398741 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":259,"threads_started":1}
I20260620 20:25:39.448980 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:39.448269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.129:43677)
0620 20:25:39.448730 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:39.448813 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":237,"threads_started":1}
I20260620 20:25:39.938385 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:39.937585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.130:42307)
0620 20:25:39.938067 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:39.938189 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":250,"threads_started":1}
I20260620 20:25:40.400731 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:40.399983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.131:54849)
0620 20:25:40.400435 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:40.400546 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":235,"threads_started":1}
I20260620 20:25:40.450666 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:40.449924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.129:37453)
0620 20:25:40.450385 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:40.450474 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":217,"threads_started":1}
I20260620 20:25:40.940073 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:40.939328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.130:49531)
0620 20:25:40.939820 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:40.939906 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":258,"threads_started":1}
I20260620 20:25:41.402307 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:41.401622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.131:49589)
0620 20:25:41.402058 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:41.402138 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":217,"threads_started":1}
I20260620 20:25:41.452276 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:41.451577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.129:39719)
0620 20:25:41.451995 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:41.452083 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":204,"threads_started":1}
I20260620 20:25:41.942016 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:41.941044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.130:40641)
0620 20:25:41.941648 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:41.941790 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":259,"threads_started":1}
I20260620 20:25:42.403915 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:42.403212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.131:46943)
0620 20:25:42.403667 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:42.403749 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":238,"threads_started":1}
I20260620 20:25:42.453807 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:42.453056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.129:37455)
0620 20:25:42.453467 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:42.453612 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":223,"threads_started":1}
I20260620 20:25:42.944308 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:42.943222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.130:58423)
0620 20:25:42.944000 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:42.944119 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":338,"threads_started":1}
I20260620 20:25:43.405651 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:43.404797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.131:38421)
0620 20:25:43.405324 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:43.405413 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":313,"threads_started":1}
I20260620 20:25:43.455472 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:43.454676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.129:59735)
0620 20:25:43.455171 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:43.455256 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":234,"threads_started":1}
I20260620 20:25:43.946206 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:43.945301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.130:35689)
0620 20:25:43.945862 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:43.945976 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":302,"threads_started":1}
I20260620 20:25:44.407393 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:44.406599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.131:35077)
0620 20:25:44.407079 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:44.407166 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":214,"threads_started":1}
I20260620 20:25:44.457156 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:44.456449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.129:38587)
0620 20:25:44.456898 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:44.456988 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":232,"threads_started":1}
I20260620 20:25:44.947875 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:44.947132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.130:44423)
0620 20:25:44.947607 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:44.947703 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":232,"threads_started":1}
I20260620 20:25:45.409597 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:45.408424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.131:36737)
0620 20:25:45.409243 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:45.409357 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":555,"threads_started":1}
I20260620 20:25:45.458930 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:45.458194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.129:56831)
0620 20:25:45.458671 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:45.458756 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":232,"threads_started":1}
I20260620 20:25:45.949801 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:45.948904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.130:36379)
0620 20:25:45.949451 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:45.949611 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":305,"threads_started":1}
I20260620 20:25:46.411339 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:46.410585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.131:37629)
0620 20:25:46.411033 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:46.411140 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":208,"threads_started":1}
I20260620 20:25:46.460677 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:46.459963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.129:52163)
0620 20:25:46.460428 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:46.460512 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":260,"threads_started":1}
I20260620 20:25:46.951504 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:46.950777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.130:50757)
0620 20:25:46.951245 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:46.951340 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":253,"threads_started":1}
I20260620 20:25:47.413102 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:47.412362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.131:56317)
0620 20:25:47.412831 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:47.412915 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":259,"threads_started":1}
I20260620 20:25:47.462500 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:47.461751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.129:44311)
0620 20:25:47.462197 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:47.462314 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":208,"threads_started":1}
I20260620 20:25:47.953198 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:47.952456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.130:46299)
0620 20:25:47.952892 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:47.952979 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":223,"threads_started":1}
I20260620 20:25:48.414932 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:48.414075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.131:34031)
0620 20:25:48.414622 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:48.414729 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":269,"threads_started":1}
I20260620 20:25:48.464227 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:48.463515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.129:45715)
0620 20:25:48.463960 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:48.464060 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":225,"threads_started":1}
I20260620 20:25:48.955071 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:48.954332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.130:51479)
0620 20:25:48.954796 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:48.954884 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":223,"threads_started":1}
I20260620 20:25:49.416747 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:49.415991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.131:41909)
0620 20:25:49.416494 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:49.416580 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":289,"threads_started":1}
I20260620 20:25:49.466691 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:49.465253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.129:58993)
0620 20:25:49.466322 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:49.466439 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":931,"thread_start_us":336,"threads_started":1}
I20260620 20:25:49.956908 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:49.956124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.130:60147)
0620 20:25:49.956634 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:49.956728 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":271,"threads_started":1}
I20260620 20:25:50.418673 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:50.417850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.131:42289)
0620 20:25:50.418379 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:50.418477 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":283,"threads_started":1}
I20260620 20:25:50.468513 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:50.467640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.129:38185)
0620 20:25:50.468143 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:50.468245 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":233,"threads_started":1}
I20260620 20:25:50.958683 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:50.957876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.130:47271)
0620 20:25:50.958349 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:50.958448 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":233,"threads_started":1}
I20260620 20:25:51.420435 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:51.419721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.131:45153)
0620 20:25:51.420165 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:51.420262 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":212,"threads_started":1}
I20260620 20:25:51.470839 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:51.469916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.129:52023)
0620 20:25:51.470473 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:51.470608 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":267,"threads_started":1}
I20260620 20:25:51.962935 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:51.962116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.130:49513)
0620 20:25:51.962591 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:51.962744 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":218,"threads_started":1}
I20260620 20:25:52.402066 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 15223
I20260620 20:25:52.420993 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:52.403282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41148 (local address 127.31.235.129:32811)
0620 20:25:52.404156 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:52.404177 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:52.404224 (+ 47us) server_negotiation.cc:407] Connection header received
0620 20:25:52.405917 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:52.405927 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:52.406361 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:52.406732 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:52.408263 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:52.409465 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:52.410789 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:52.411483 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:52.411682 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:52.414566 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:52.414574 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:52.414590 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:52.414633 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:52.417451 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:52.418319 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:52.418325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:52.418332 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:52.418494 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:52.418889 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:52.418896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:52.418899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:52.419472 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:52.419779 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:52.420260 (+ 481us) server_negotiation.cc:299] Negotiation successful
0620 20:25:52.420691 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":458,"threads_started":1}
I20260620 20:25:52.422312 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:52.421557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.131:54901)
0620 20:25:52.422009 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:52.422132 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":217,"threads_started":1}
I20260620 20:25:52.422727 15380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:52.509743 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:52.508577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46879 (local address 127.31.235.129:50919)
0620 20:25:52.509157 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:52.509391 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:46879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":272,"threads_started":1}
I20260620 20:25:52.510042 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 15223
I20260620 20:25:52.528061 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 15388
I20260620 20:25:52.546322 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:52.529090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.31.235.130:35781)
0620 20:25:52.529721 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:52.529735 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:52.529772 (+ 37us) server_negotiation.cc:407] Connection header received
0620 20:25:52.531346 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:52.531356 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:52.531791 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:52.532146 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:52.533620 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:52.534913 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:52.536192 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:52.536865 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:52.537071 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:52.539849 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:52.539857 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:52.539873 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:52.539913 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:52.542828 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:52.543606 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:52.543614 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:52.543620 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:52.543795 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:52.544211 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:52.544216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:52.544220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:52.544768 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:52.545077 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:52.545569 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 20:25:52.546007 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":263,"threads_started":1}
I20260620 20:25:52.547811 15511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:52.632131 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 15388
I20260620 20:25:52.649716 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 15519
I20260620 20:25:52.667567 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:52.650705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44702 (local address 127.31.235.131:42689)
0620 20:25:52.651355 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:52.651370 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:52.651410 (+ 40us) server_negotiation.cc:407] Connection header received
0620 20:25:52.652999 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:52.653008 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:52.653418 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:52.653819 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:52.655210 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:52.656429 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:52.657820 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:52.658463 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:52.658659 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:52.661321 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:52.661328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:52.661344 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:52.661380 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:52.664220 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:52.664916 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:52.664928 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:52.664935 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:52.665087 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:52.665475 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:52.665483 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:52.665510 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:52.666039 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:52.666357 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:52.666853 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 20:25:52.667264 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":345,"threads_started":1}
I20260620 20:25:52.668980 15642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:52.752238 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 15519
2026-06-20T20:25:52Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24610 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:25:52 dist-test-slave-zzwf krb5kdc[15705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:25:52 dist-test-slave-zzwf krb5kdc[15705](info): set up 2 sockets
Jun 20 20:25:52 dist-test-slave-zzwf krb5kdc[15705](info): commencing operation
krb5kdc: starting...
W20260620 20:25:53.866483 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:25:54 dist-test-slave-zzwf krb5kdc[15705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:25:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:25:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:25:54.110148 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:39161
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:39161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:25:54.442785 15718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:54.443085 15718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:54.443181 15718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:54.450542 15718 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 20:25:54.450651 15718 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 20:25:54.450695 15718 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 20:25:54.450742 15718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:25:54.450799 15718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:54.450868 15718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:54.450914 15718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:25:54.450961 15718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:25:54.460948 15718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:39161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:39161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:54.462251 15718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:54.463546 15718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:54.475356 15723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:54.475384 15726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:54.475329 15724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:54.475996 15718 server_base.cc:1034] running on GCE node
I20260620 20:25:54.476783 15718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:54.478204 15718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:54.479471 15718 hybrid_clock.cc:648] HybridClock initialized: now 1781987154479402 us; error 47 us; skew 500 ppm
I20260620 20:25:54.479817 15718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:54 dist-test-slave-zzwf krb5kdc[15705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987154, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:54.484368 15718 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:25:54.485942 15718 webserver.cc:460] Webserver started at http://127.31.235.190:43605/ using document root <none> and password file <none>
I20260620 20:25:54.486647 15718 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:54.486779 15718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:54.487097 15718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:25:54.490100 15718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "37f76da362154684a3f612ac76e371b6"
format_stamp: "Formatted at 2026-06-20 20:25:54 on dist-test-slave-zzwf"
server_key: "86f2f19ac948bf51a8a7c5ed711070d3"
server_key_iv: "c0daea5a53a7ec262d3abacbd87ab597"
server_key_version: "encryptionkey@0"
I20260620 20:25:54.490943 15718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "37f76da362154684a3f612ac76e371b6"
format_stamp: "Formatted at 2026-06-20 20:25:54 on dist-test-slave-zzwf"
server_key: "86f2f19ac948bf51a8a7c5ed711070d3"
server_key_iv: "c0daea5a53a7ec262d3abacbd87ab597"
server_key_version: "encryptionkey@0"
I20260620 20:25:54.496030 15718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:25:54.499487 15733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:54.500823 15718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:25:54.501012 15718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "37f76da362154684a3f612ac76e371b6"
format_stamp: "Formatted at 2026-06-20 20:25:54 on dist-test-slave-zzwf"
server_key: "86f2f19ac948bf51a8a7c5ed711070d3"
server_key_iv: "c0daea5a53a7ec262d3abacbd87ab597"
server_key_version: "encryptionkey@0"
I20260620 20:25:54.501250 15718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:54.520866 15718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:54.525390 15718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:54.525729 15718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:54.545506 15718 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:39161
I20260620 20:25:54.545558 15784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:39161 every 8 connection(s)
I20260620 20:25:54.546939 15718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:25:54.550922 15785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:25:54.553762 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 15718
I20260620 20:25:54.553967 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:25:54.554733 32686 external_mini_cluster.cc:1428] Setting key acd8dbb0e362957b828defc75b3a5af9
I20260620 20:25:54.564054 15785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6: Bootstrap starting.
I20260620 20:25:54.567983 15785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6: Neither blocks nor log segments found. Creating new log.
I20260620 20:25:54.569336 15785 log.cc:824] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:25:54 dist-test-slave-zzwf krb5kdc[15705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:25:54.572268 15785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6: No bootstrap required, opened a new log
I20260620 20:25:54.578466 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:54.556875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44794 (local address 127.31.235.190:39161)
0620 20:25:54.557739 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:54.557753 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:54.557794 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:25:54.559468 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:54.559491 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:54.559867 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:54.560189 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:54.562060 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:54.563378 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:54.565248 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:54.565991 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:54.566119 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:54.571145 (+ 5026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:54.571153 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:54.571167 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:54.571205 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:54.574353 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:54.575144 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:54.575151 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:54.575157 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:54.575319 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:54.575703 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:54.575710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:54.575713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:54.576253 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:54.576589 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:54.576996 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 20:25:54.577419 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":413,"threads_started":1}
I20260620 20:25:54.580304 15785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f76da362154684a3f612ac76e371b6" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39161 } }
I20260620 20:25:54.580744 15785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:25:54.580857 15785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37f76da362154684a3f612ac76e371b6, State: Initialized, Role: FOLLOWER
I20260620 20:25:54.581490 15785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f76da362154684a3f612ac76e371b6" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39161 } }
I20260620 20:25:54.581770 15785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:25:54.581873 15785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:25:54.581986 15785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:25:54.584309 15785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f76da362154684a3f612ac76e371b6" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39161 } }
I20260620 20:25:54.584865 15785 leader_election.cc:302] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37f76da362154684a3f612ac76e371b6; no voters:
I20260620 20:25:54.585369 15785 leader_election.cc:288] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:25:54.585531 15790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:25:54.586298 15790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 1 LEADER]: Becoming Leader. State: Replica: 37f76da362154684a3f612ac76e371b6, State: Running, Role: LEADER
I20260620 20:25:54.586778 15790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f76da362154684a3f612ac76e371b6" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39161 } }
I20260620 20:25:54.587626 15785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:25:54.589728 15791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37f76da362154684a3f612ac76e371b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f76da362154684a3f612ac76e371b6" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39161 } } }
I20260620 20:25:54.589758 15792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37f76da362154684a3f612ac76e371b6. Latest consensus state: current_term: 1 leader_uuid: "37f76da362154684a3f612ac76e371b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f76da362154684a3f612ac76e371b6" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39161 } } }
I20260620 20:25:54.590193 15792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [sys.catalog]: This master's current role is: LEADER
I20260620 20:25:54.590219 15791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [sys.catalog]: This master's current role is: LEADER
I20260620 20:25:54.592383 15796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:25:54.597875 15796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:25:54.606948 15796 catalog_manager.cc:1347] Generated new cluster ID: a44ca722a80f40aabbcf41b0c8ad76d7
I20260620 20:25:54.607064 15796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:25:54.622125 15796 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:25:54.623426 15796 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:25:54.634676 15796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6: Generated new TSK 0
I20260620 20:25:54.635586 15796 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
I20260620 20:25:54.694101 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:39161
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:25:55.027594 15811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:55.027885 15811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:55.028024 15811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:55.035483 15811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:55.035600 15811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:55.035780 15811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:25:55.046095 15811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:39161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:55.047497 15811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:55.048960 15811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:55.062508 15817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:55.062594 15816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:55.062618 15819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:55.063627 15811 server_base.cc:1034] running on GCE node
I20260620 20:25:55.064281 15811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:55.065258 15811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:55.066483 15811 hybrid_clock.cc:648] HybridClock initialized: now 1781987155066398 us; error 71 us; skew 500 ppm
I20260620 20:25:55.066828 15811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:55 dist-test-slave-zzwf krb5kdc[15705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987155, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:55.070144 15811 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:25:55.072137 15811 webserver.cc:460] Webserver started at http://127.31.235.129:35663/ using document root <none> and password file <none>
I20260620 20:25:55.072847 15811 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:55.072956 15811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:55.073223 15811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:25:55.076387 15811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "225f81a50ceb4492b32d74188461b59f"
format_stamp: "Formatted at 2026-06-20 20:25:55 on dist-test-slave-zzwf"
server_key: "4488c6bbdda90d61cd5a939548c80705"
server_key_iv: "8b1e017e2e26aca057ad3eea9628f09a"
server_key_version: "encryptionkey@0"
I20260620 20:25:55.077122 15811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "225f81a50ceb4492b32d74188461b59f"
format_stamp: "Formatted at 2026-06-20 20:25:55 on dist-test-slave-zzwf"
server_key: "4488c6bbdda90d61cd5a939548c80705"
server_key_iv: "8b1e017e2e26aca057ad3eea9628f09a"
server_key_version: "encryptionkey@0"
I20260620 20:25:55.083180 15811 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 20:25:55.087141 15826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:55.088912 15811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:25:55.089154 15811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "225f81a50ceb4492b32d74188461b59f"
format_stamp: "Formatted at 2026-06-20 20:25:55 on dist-test-slave-zzwf"
server_key: "4488c6bbdda90d61cd5a939548c80705"
server_key_iv: "8b1e017e2e26aca057ad3eea9628f09a"
server_key_version: "encryptionkey@0"
I20260620 20:25:55.089367 15811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:55.116037 15811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:55.120565 15811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:55.120864 15811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:55.121882 15811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:25:55.123214 15811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:25:55.123286 15811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:55.123430 15811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:25:55.123495 15811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:55.156842 15811 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:45307
I20260620 20:25:55.156898 15938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:45307 every 8 connection(s)
I20260620 20:25:55.158396 15811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:25:55.161690 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 15811
I20260620 20:25:55.161921 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:25:55.162746 32686 external_mini_cluster.cc:1428] Setting key 6ea2ec91f783274be770b9bf62e22d2f
Jun 20 20:25:55 dist-test-slave-zzwf krb5kdc[15705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987155, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:25:55.179690 15941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:55.161010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:42303 (local address 127.31.235.190:39161)
0620 20:25:55.161572 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:55.161577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:55.162795 (+ 1218us) server_negotiation.cc:407] Connection header received
0620 20:25:55.164970 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:55.164976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:55.165069 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:55.165198 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:55.166808 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:55.167544 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:55.169133 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:55.169620 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:55.169762 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:55.173244 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:55.173250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:55.173255 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:55.173296 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:55.176189 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:55.177044 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:55.177050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:55.177054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:55.177195 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:55.177737 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:55.177744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:55.177748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:55.178053 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:55.178239 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:55.179028 (+ 789us) server_negotiation.cc:299] Negotiation successful
0620 20:25:55.179297 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":2094,"threads_started":1}
I20260620 20:25:55.180783 15942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:55.161695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:42303)
0620 20:25:55.162541 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:55.162586 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:55.164706 (+ 2120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:55.165445 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:55.165454 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:55.166048 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:55.166627 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:55.166645 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:55.167703 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:55.167709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:55.168944 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:55.168957 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:55.169852 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:55.169861 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:55.170110 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:55.170833 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:55.170855 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:55.173019 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:55.176372 (+ 3353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:55.176384 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:55.176411 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:55.176894 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:55.177317 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:55.177322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:55.177327 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:55.177618 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:55.178209 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:55.178218 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:55.178609 (+ 391us) client_negotiation.cc:769] Sending connection context
0620 20:25:55.178904 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 20:25:55.179485 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":379,"threads_started":1}
I20260620 20:25:55.182672 15939 heartbeater.cc:347] Connected to a master server at 127.31.235.190:39161
I20260620 20:25:55.183041 15939 heartbeater.cc:464] Registering TS with master...
I20260620 20:25:55.183997 15939 heartbeater.cc:511] Master 127.31.235.190:39161 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.235.130@KRBTEST.COM; defaulting to no policy
I20260620 20:25:55.186681 15750 ts_manager.cc:194] Registered new tserver with Master: 225f81a50ceb4492b32d74188461b59f (127.31.235.129:45307)
I20260620 20:25:55.188301 15750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:42303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
I20260620 20:25:55.220393 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:39161
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:25:55.555970 15945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:55.556228 15945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:55.556411 15945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:55.563702 15945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:55.563804 15945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:55.563982 15945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:25:55.574051 15945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:39161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:55.575380 15945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:55.576697 15945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:55.590315 15950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:55.590474 15951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:55.590474 15953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:55.591759 15945 server_base.cc:1034] running on GCE node
I20260620 20:25:55.592589 15945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:55.593806 15945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:55.595047 15945 hybrid_clock.cc:648] HybridClock initialized: now 1781987155594970 us; error 75 us; skew 500 ppm
I20260620 20:25:55.595399 15945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:55 dist-test-slave-zzwf krb5kdc[15705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987155, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:55.598676 15945 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:25:55.600522 15945 webserver.cc:460] Webserver started at http://127.31.235.130:43767/ using document root <none> and password file <none>
I20260620 20:25:55.601287 15945 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:55.601437 15945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:55.601802 15945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:25:55.605022 15945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "1e513437eeac457aa8354f95edcbd9ec"
format_stamp: "Formatted at 2026-06-20 20:25:55 on dist-test-slave-zzwf"
server_key: "fb5e88801ae36d4fe06b819cf2966509"
server_key_iv: "0d924090dcd9c20b8de5b97fce19af5b"
server_key_version: "encryptionkey@0"
I20260620 20:25:55.605938 15945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "1e513437eeac457aa8354f95edcbd9ec"
format_stamp: "Formatted at 2026-06-20 20:25:55 on dist-test-slave-zzwf"
server_key: "fb5e88801ae36d4fe06b819cf2966509"
server_key_iv: "0d924090dcd9c20b8de5b97fce19af5b"
server_key_version: "encryptionkey@0"
I20260620 20:25:55.611234 15945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:25:55.614853 15960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:55.616313 15945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:25:55.616534 15945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "1e513437eeac457aa8354f95edcbd9ec"
format_stamp: "Formatted at 2026-06-20 20:25:55 on dist-test-slave-zzwf"
server_key: "fb5e88801ae36d4fe06b819cf2966509"
server_key_iv: "0d924090dcd9c20b8de5b97fce19af5b"
server_key_version: "encryptionkey@0"
I20260620 20:25:55.616779 15945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:55.638548 15945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:55.642858 15945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:55.643189 15945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:55.644218 15945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:25:55.645663 15945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:25:55.645776 15945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:55.645941 15945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:25:55.646023 15945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:55.681336 15945 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:36427
I20260620 20:25:55.681380 16072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:36427 every 8 connection(s)
I20260620 20:25:55.682897 15945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:25:55.684391 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 15945
I20260620 20:25:55.684638 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:25:55.685182 32686 external_mini_cluster.cc:1428] Setting key d174a2aa30c94765ca41abb6d8bc4f23
Jun 20 20:25:55 dist-test-slave-zzwf krb5kdc[15705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987155, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:25:55.708460 16075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:55.685676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:35715 (local address 127.31.235.190:39161)
0620 20:25:55.686198 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:55.686204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:55.687562 (+ 1358us) server_negotiation.cc:407] Connection header received
0620 20:25:55.690796 (+ 3234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:55.690800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:55.690869 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:55.690962 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:55.693391 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:55.694543 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:55.697361 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:55.697859 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:55.697956 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:55.702427 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:55.702432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:55.702436 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:55.702470 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:55.705316 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:55.706117 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:55.706125 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:55.706130 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:55.706268 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:55.706703 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:55.706708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:55.706711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:55.706990 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:55.707147 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:55.707846 (+ 699us) server_negotiation.cc:299] Negotiation successful
0620 20:25:55.708079 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":253,"threads_started":1}
I20260620 20:25:55.709632 16076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:55.686355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.130:35715)
0620 20:25:55.687388 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:55.687447 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:55.690481 (+ 3034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:55.691265 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:55.691280 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:55.692152 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:55.693121 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:55.693151 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:55.694735 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:55.694744 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:55.697192 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:55.697214 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:55.698119 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:55.698128 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:55.698488 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:55.699233 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:55.699274 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:55.702148 (+ 2874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:55.705452 (+ 3304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:55.705480 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:55.705519 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:55.705968 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:55.706382 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:55.706388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:55.706392 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:55.706580 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:55.707115 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:55.707124 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:55.707477 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 20:25:55.707752 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 20:25:55.708299 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":352,"threads_started":1}
I20260620 20:25:55.711118 16073 heartbeater.cc:347] Connected to a master server at 127.31.235.190:39161
I20260620 20:25:55.711444 16073 heartbeater.cc:464] Registering TS with master...
I20260620 20:25:55.712160 16073 heartbeater.cc:511] Master 127.31.235.190:39161 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.235.131@KRBTEST.COM; defaulting to no policy
I20260620 20:25:55.713685 15750 ts_manager.cc:194] Registered new tserver with Master: 1e513437eeac457aa8354f95edcbd9ec (127.31.235.130:36427)
I20260620 20:25:55.714573 15750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:35715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
I20260620 20:25:55.744767 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:39161
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:25:56.070492 16079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:25:56.070879 16079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:25:56.071065 16079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:25:56.078817 16079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:25:56.079005 16079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:25:56.079202 16079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:25:56.089777 16079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:39161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:25:56.091127 16079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:25:56.092476 16079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:25:56.106451 16084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:56.106509 16085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:25:56.106412 16087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:25:56.107554 16079 server_base.cc:1034] running on GCE node
I20260620 20:25:56.108212 16079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:25:56.109403 16079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:25:56.110703 16079 hybrid_clock.cc:648] HybridClock initialized: now 1781987156110634 us; error 59 us; skew 500 ppm
I20260620 20:25:56.111081 16079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:25:56 dist-test-slave-zzwf krb5kdc[15705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987156, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:25:56.114826 16079 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:25:56.116431 16079 webserver.cc:460] Webserver started at http://127.31.235.131:46867/ using document root <none> and password file <none>
I20260620 20:25:56.117169 16079 fs_manager.cc:359] Metadata directory not provided
I20260620 20:25:56.117303 16079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:25:56.117734 16079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:25:56.120800 16079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "f4de56503aca4b3096531ba2ea1a30d5"
format_stamp: "Formatted at 2026-06-20 20:25:56 on dist-test-slave-zzwf"
server_key: "2a794319c1cdb4011cca3c54091d52cf"
server_key_iv: "7ca62692cd477ae84f3f61e22ec8946d"
server_key_version: "encryptionkey@0"
I20260620 20:25:56.121763 16079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "f4de56503aca4b3096531ba2ea1a30d5"
format_stamp: "Formatted at 2026-06-20 20:25:56 on dist-test-slave-zzwf"
server_key: "2a794319c1cdb4011cca3c54091d52cf"
server_key_iv: "7ca62692cd477ae84f3f61e22ec8946d"
server_key_version: "encryptionkey@0"
I20260620 20:25:56.127549 16079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:25:56.131223 16094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:56.132705 16079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:25:56.132925 16079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "f4de56503aca4b3096531ba2ea1a30d5"
format_stamp: "Formatted at 2026-06-20 20:25:56 on dist-test-slave-zzwf"
server_key: "2a794319c1cdb4011cca3c54091d52cf"
server_key_iv: "7ca62692cd477ae84f3f61e22ec8946d"
server_key_version: "encryptionkey@0"
I20260620 20:25:56.133147 16079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:25:56.149391 16079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:25:56.154227 16079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:25:56.154563 16079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:25:56.155648 16079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:25:56.157299 16079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:25:56.157380 16079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:56.157569 16079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:25:56.157630 16079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:25:56.191437 15939 heartbeater.cc:503] Master 127.31.235.190:39161 was elected leader, sending a full tablet report...
I20260620 20:25:56.192752 16079 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:36079
I20260620 20:25:56.192827 16206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:36079 every 8 connection(s)
I20260620 20:25:56.194478 16079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:25:56.198248 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 16079
I20260620 20:25:56.198477 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:25:56.199034 32686 external_mini_cluster.cc:1428] Setting key 00536933ebe79e2b36e0167e233778e5
Jun 20 20:25:56 dist-test-slave-zzwf krb5kdc[15705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987156, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:25:56.215476 16075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:56.197114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:54615 (local address 127.31.235.190:39161)
0620 20:25:56.197336 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:56.197343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:56.198700 (+ 1357us) server_negotiation.cc:407] Connection header received
0620 20:25:56.200843 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:56.200847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:56.200970 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:56.201087 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:56.202984 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.203716 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.205284 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.206045 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.206155 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:56.209426 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:56.209431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:56.209434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:56.209466 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:56.212189 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:56.212941 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:56.212947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:56.212950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:56.213091 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:56.213633 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:56.213638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:56.213641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:56.214021 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:56.214194 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:56.215054 (+ 860us) server_negotiation.cc:299] Negotiation successful
0620 20:25:56.215227 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 20:25:56.216481 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:56.197696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.131:54615)
0620 20:25:56.198529 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:56.198573 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:56.200567 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:56.201395 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:56.201407 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:56.202254 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:25:56.202831 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:56.202848 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.203842 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.203846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:56.205139 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:56.205151 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.206243 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.206248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:56.206448 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:56.207084 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:25:56.207105 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:25:56.209235 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:25:56.212313 (+ 3078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:56.212319 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:56.212333 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:56.212799 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:56.213227 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:25:56.213231 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:25:56.213235 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:25:56.213497 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:25:56.214177 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:25:56.214184 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:25:56.214640 (+ 456us) client_negotiation.cc:769] Sending connection context
0620 20:25:56.214941 (+ 301us) client_negotiation.cc:240] Negotiation successful
0620 20:25:56.215332 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":357,"threads_started":1}
I20260620 20:25:56.217823 16207 heartbeater.cc:347] Connected to a master server at 127.31.235.190:39161
I20260620 20:25:56.218130 16207 heartbeater.cc:464] Registering TS with master...
I20260620 20:25:56.218858 16207 heartbeater.cc:511] Master 127.31.235.190:39161 requested a full tablet report, sending...
I20260620 20:25:56.220378 15750 ts_manager.cc:194] Registered new tserver with Master: f4de56503aca4b3096531ba2ea1a30d5 (127.31.235.131:36079)
I20260620 20:25:56.221268 15750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:54615
I20260620 20:25:56.227669 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:25:56.246311 16075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:56.233052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44806 (local address 127.31.235.190:39161)
0620 20:25:56.233332 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:56.233341 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:56.233761 (+ 420us) server_negotiation.cc:407] Connection header received
0620 20:25:56.234007 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:56.234011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:56.234077 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:56.234187 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:56.235628 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.236424 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.237997 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.238489 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.238638 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:56.240973 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:56.240982 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:56.240991 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:56.241022 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:56.243596 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:56.244265 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:56.244277 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:56.244280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:56.244393 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:56.244768 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:56.244774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:56.244777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:56.245081 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:56.245356 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:56.245737 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 20:25:56.245973 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 20:25:56.251936 15750 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:25:56.263461 15750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:25:56.289316 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:56.276848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:45307 (local address 127.0.0.1:40642)
0620 20:25:56.277574 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:56.277624 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:56.277908 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:56.279026 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:56.279044 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:56.279089 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:56.279775 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:56.279795 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.282665 (+ 2870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.282672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:56.285739 (+ 3067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:56.285757 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.287764 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.287773 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:56.288060 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:56.288269 (+ 209us) client_negotiation.cc:769] Sending connection context
0620 20:25:56.288499 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 20:25:56.288728 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":25,"spinlock_wait_cycles":3328,"thread_start_us":422,"threads_started":1}
I20260620 20:25:56.290526 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:56.278208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:36079 (local address 127.0.0.1:47518)
0620 20:25:56.278739 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:56.278762 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:56.278862 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:56.282916 (+ 4054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:56.282923 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:56.282955 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:56.283658 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:56.283678 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.285801 (+ 2123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.285806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:56.288233 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:56.288244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.289679 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.289687 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:56.289871 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:56.289895 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:25:56.289950 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 20:25:56.290040 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":1,"thread_start_us":270,"threads_started":1}
I20260620 20:25:56.290895 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:56.277569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40642 (local address 127.31.235.129:45307)
0620 20:25:56.278238 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:56.278245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:56.278270 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:25:56.278405 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:56.278415 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:56.278716 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:56.280052 (+ 1336us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:56.280351 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.282473 (+ 2122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.285968 (+ 3495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.287486 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.290008 (+ 2522us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:56.290244 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:56.290323 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 20:25:56.290521 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":354,"threads_started":1}
I20260620 20:25:56.291287 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:56.278845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:36427 (local address 127.0.0.1:46854)
0620 20:25:56.279486 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:56.279509 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:56.279622 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:56.282621 (+ 2999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:56.282630 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:56.282665 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:56.284174 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:56.284183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.285466 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.285472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:56.287868 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:56.287883 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.290282 (+ 2399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.290289 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:56.290598 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:56.290641 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 20:25:56.290741 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 20:25:56.290892 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":76,"thread_start_us":669,"threads_started":1}
I20260620 20:25:56.292351 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:56.278568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47518 (local address 127.31.235.131:36079)
0620 20:25:56.282351 (+ 3783us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:56.282359 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:56.282390 (+ 31us) server_negotiation.cc:407] Connection header received
0620 20:25:56.282484 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:56.282496 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:56.282693 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:56.283144 (+ 451us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:56.283890 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.285652 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.288384 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.289382 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.290944 (+ 1562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:56.291739 (+ 795us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:56.291815 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 20:25:56.291975 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3632,"thread_start_us":3473,"threads_started":1}
I20260620 20:25:56.292430 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:56.279357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46854 (local address 127.31.235.130:36427)
0620 20:25:56.281959 (+ 2602us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:56.281970 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:56.282002 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:25:56.282115 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:56.282124 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:56.282395 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:56.282988 (+ 593us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:56.284331 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.285312 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.288174 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.290064 (+ 1890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.290241 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:56.290408 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:56.291893 (+ 1485us) server_negotiation.cc:299] Negotiation successful
0620 20:25:56.292050 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2444,"thread_start_us":2319,"threads_started":1}
I20260620 20:25:56.295560 16008 tablet_service.cc:1467] Processing CreateTablet for tablet 2aac9f6cdfb84625b5b3438ae6e18d75 (DEFAULT_TABLE table=test-table [id=c20e29af234d4ce99604f2f26c32618d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:25:56.296095 16142 tablet_service.cc:1467] Processing CreateTablet for tablet 2aac9f6cdfb84625b5b3438ae6e18d75 (DEFAULT_TABLE table=test-table [id=c20e29af234d4ce99604f2f26c32618d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:25:56.295471 15874 tablet_service.cc:1467] Processing CreateTablet for tablet 2aac9f6cdfb84625b5b3438ae6e18d75 (DEFAULT_TABLE table=test-table [id=c20e29af234d4ce99604f2f26c32618d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:25:56.296885 16008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aac9f6cdfb84625b5b3438ae6e18d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:25:56.297622 16142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aac9f6cdfb84625b5b3438ae6e18d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:25:56.297950 15874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aac9f6cdfb84625b5b3438ae6e18d75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:25:56.308327 16225 tablet_bootstrap.cc:492] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec: Bootstrap starting.
I20260620 20:25:56.311638 16225 tablet_bootstrap.cc:654] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec: Neither blocks nor log segments found. Creating new log.
I20260620 20:25:56.312665 16225 log.cc:824] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec: Log is configured to *not* fsync() on all Append() calls
I20260620 20:25:56.313629 16226 tablet_bootstrap.cc:492] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5: Bootstrap starting.
I20260620 20:25:56.318552 16228 tablet_bootstrap.cc:492] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f: Bootstrap starting.
I20260620 20:25:56.318755 16226 tablet_bootstrap.cc:654] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5: Neither blocks nor log segments found. Creating new log.
I20260620 20:25:56.320061 16226 log.cc:824] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5: Log is configured to *not* fsync() on all Append() calls
I20260620 20:25:56.321247 16225 tablet_bootstrap.cc:492] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec: No bootstrap required, opened a new log
I20260620 20:25:56.321564 16225 ts_tablet_manager.cc:1397] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20260620 20:25:56.322909 16226 tablet_bootstrap.cc:492] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5: No bootstrap required, opened a new log
I20260620 20:25:56.323205 16226 ts_tablet_manager.cc:1397] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 20:25:56.323714 16228 tablet_bootstrap.cc:654] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f: Neither blocks nor log segments found. Creating new log.
I20260620 20:25:56.324905 16228 log.cc:824] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:25:56.327849 16228 tablet_bootstrap.cc:492] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f: No bootstrap required, opened a new log
I20260620 20:25:56.328104 16228 ts_tablet_manager.cc:1397] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260620 20:25:56.331974 16226 raft_consensus.cc:348] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } }
I20260620 20:25:56.332019 16225 raft_consensus.cc:348] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } }
I20260620 20:25:56.332392 16226 raft_consensus.cc:374] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:25:56.332439 16225 raft_consensus.cc:374] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:25:56.332487 16226 raft_consensus.cc:729] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4de56503aca4b3096531ba2ea1a30d5, State: Initialized, Role: FOLLOWER
I20260620 20:25:56.332535 16225 raft_consensus.cc:729] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e513437eeac457aa8354f95edcbd9ec, State: Initialized, Role: FOLLOWER
I20260620 20:25:56.333132 16226 consensus_queue.cc:260] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } }
I20260620 20:25:56.333170 16225 consensus_queue.cc:260] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } }
I20260620 20:25:56.334369 16207 heartbeater.cc:503] Master 127.31.235.190:39161 was elected leader, sending a full tablet report...
I20260620 20:25:56.334990 16226 ts_tablet_manager.cc:1428] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260620 20:25:56.340334 16228 raft_consensus.cc:348] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } }
I20260620 20:25:56.340962 16228 raft_consensus.cc:374] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:25:56.341084 16228 raft_consensus.cc:729] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 225f81a50ceb4492b32d74188461b59f, State: Initialized, Role: FOLLOWER
I20260620 20:25:56.341813 16228 consensus_queue.cc:260] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } }
I20260620 20:25:56.342855 16225 ts_tablet_manager.cc:1428] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec: Time spent starting tablet: real 0.021s user 0.010s sys 0.008s
I20260620 20:25:56.343461 16073 heartbeater.cc:503] Master 127.31.235.190:39161 was elected leader, sending a full tablet report...
I20260620 20:25:56.344971 16228 ts_tablet_manager.cc:1428] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f: Time spent starting tablet: real 0.017s user 0.015s sys 0.002s
W20260620 20:25:56.410879 15940 tablet.cc:2369] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:25:56.435153 16074 tablet.cc:2369] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:25:56.446244 16208 tablet.cc:2369] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:25:56.683228 16231 raft_consensus.cc:482] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:25:56.683522 16231 raft_consensus.cc:504] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } }
I20260620 20:25:56.685530 16231 leader_election.cc:288] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 225f81a50ceb4492b32d74188461b59f (127.31.235.129:45307), 1e513437eeac457aa8354f95edcbd9ec (127.31.235.130:36427)
I20260620 20:25:56.692860 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:56.685963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:45307 (local address 127.31.235.131:47585)
0620 20:25:56.686340 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:56.686359 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:56.686501 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:56.687027 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:56.687032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:56.687057 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:56.687260 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:56.687269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.688718 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.688723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:56.690565 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:56.690585 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.692092 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.692098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:56.692304 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:56.692342 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 20:25:56.692444 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 20:25:56.692589 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264}
I20260620 20:25:56.692875 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:56.685963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:47585 (local address 127.31.235.129:45307)
0620 20:25:56.686185 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:56.686191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:56.686609 (+ 418us) server_negotiation.cc:407] Connection header received
0620 20:25:56.686695 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:56.686700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:56.686771 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:56.686845 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:56.687411 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.688604 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.690734 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.691939 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.692155 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:56.692235 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:56.692484 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 20:25:56.692619 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:25:56.694016 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:56.686397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:36427 (local address 127.31.235.131:34359)
0620 20:25:56.687404 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:56.687421 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:56.687557 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:56.687924 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:56.687929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:56.687959 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:56.688279 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:56.688289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.689728 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.689734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:56.691329 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:56.691339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.693209 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.693214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:56.693387 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:56.693426 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 20:25:56.693586 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 20:25:56.693715 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"thread_start_us":723,"threads_started":1}
I20260620 20:25:56.694005 15894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aac9f6cdfb84625b5b3438ae6e18d75" candidate_uuid: "f4de56503aca4b3096531ba2ea1a30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "225f81a50ceb4492b32d74188461b59f" is_pre_election: true
I20260620 20:25:56.694350 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:56.686691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:34359 (local address 127.31.235.130:36427)
0620 20:25:56.686935 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:56.686944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:56.687523 (+ 579us) server_negotiation.cc:407] Connection header received
0620 20:25:56.687700 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:56.687706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:56.687807 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:56.687890 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:56.688438 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.689587 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.691478 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.692588 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.692778 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:56.692901 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:56.693636 (+ 735us) server_negotiation.cc:299] Negotiation successful
0620 20:25:56.693774 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 20:25:56.694577 15894 raft_consensus.cc:2454] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4de56503aca4b3096531ba2ea1a30d5 in term 0.
I20260620 20:25:56.695012 16232 raft_consensus.cc:482] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:25:56.695291 16095 leader_election.cc:302] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 225f81a50ceb4492b32d74188461b59f, f4de56503aca4b3096531ba2ea1a30d5; no voters:
I20260620 20:25:56.695312 16232 raft_consensus.cc:504] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } }
I20260620 20:25:56.695389 16028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aac9f6cdfb84625b5b3438ae6e18d75" candidate_uuid: "f4de56503aca4b3096531ba2ea1a30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e513437eeac457aa8354f95edcbd9ec" is_pre_election: true
I20260620 20:25:56.695724 16231 raft_consensus.cc:2781] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:25:56.695852 16231 raft_consensus.cc:482] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:25:56.695930 16231 raft_consensus.cc:3037] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:25:56.696429 16028 raft_consensus.cc:2454] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4de56503aca4b3096531ba2ea1a30d5 in term 0.
I20260620 20:25:56.697111 16232 leader_election.cc:288] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 225f81a50ceb4492b32d74188461b59f (127.31.235.129:45307), f4de56503aca4b3096531ba2ea1a30d5 (127.31.235.131:36079)
I20260620 20:25:56.699420 16231 raft_consensus.cc:504] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } }
I20260620 20:25:56.700435 15894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aac9f6cdfb84625b5b3438ae6e18d75" candidate_uuid: "f4de56503aca4b3096531ba2ea1a30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "225f81a50ceb4492b32d74188461b59f"
I20260620 20:25:56.700644 15894 raft_consensus.cc:3037] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:25:56.700685 16028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aac9f6cdfb84625b5b3438ae6e18d75" candidate_uuid: "f4de56503aca4b3096531ba2ea1a30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e513437eeac457aa8354f95edcbd9ec"
I20260620 20:25:56.700892 16028 raft_consensus.cc:3037] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:25:56.703944 15894 raft_consensus.cc:2454] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4de56503aca4b3096531ba2ea1a30d5 in term 1.
I20260620 20:25:56.704347 16095 leader_election.cc:302] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 225f81a50ceb4492b32d74188461b59f, f4de56503aca4b3096531ba2ea1a30d5; no voters:
I20260620 20:25:56.705282 16028 raft_consensus.cc:2454] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4de56503aca4b3096531ba2ea1a30d5 in term 1.
I20260620 20:25:56.705405 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:56.697719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:36287 (local address 127.31.235.129:45307)
0620 20:25:56.697879 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:56.697884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:56.697963 (+ 79us) server_negotiation.cc:407] Connection header received
0620 20:25:56.698158 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:56.698163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:56.698233 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:56.698335 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:56.698947 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.700053 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.702877 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.704191 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.704350 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:56.704424 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:56.704972 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 20:25:56.705110 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:25:56.705755 16237 raft_consensus.cc:2781] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:25:56.705855 16231 leader_election.cc:288] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 [CANDIDATE]: Term 1 election: Requested vote from peers 225f81a50ceb4492b32d74188461b59f (127.31.235.129:45307), 1e513437eeac457aa8354f95edcbd9ec (127.31.235.130:36427)
I20260620 20:25:56.706271 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:56.697243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:45307 (local address 127.31.235.130:36287)
0620 20:25:56.697854 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:56.697875 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:56.698013 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:56.698358 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:56.698364 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:56.698389 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:56.698796 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:56.698807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.700203 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.700207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:56.702676 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:56.702691 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.704562 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.704566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:56.704759 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:56.704807 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 20:25:56.705651 (+ 844us) client_negotiation.cc:240] Negotiation successful
0620 20:25:56.705805 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":42,"thread_start_us":552,"threads_started":1}
I20260620 20:25:56.706470 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:56.698524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:46717 (local address 127.31.235.131:36079)
0620 20:25:56.698709 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:56.698715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:56.699082 (+ 367us) server_negotiation.cc:407] Connection header received
0620 20:25:56.699249 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:56.699257 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:56.699334 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:56.699442 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:25:56.699979 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.701026 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.703805 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.705145 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.705289 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:56.705351 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:56.706105 (+ 754us) server_negotiation.cc:299] Negotiation successful
0620 20:25:56.706243 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 20:25:56.706866 16237 raft_consensus.cc:686] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 [term 1 LEADER]: Becoming Leader. State: Replica: f4de56503aca4b3096531ba2ea1a30d5, State: Running, Role: LEADER
I20260620 20:25:56.707003 15894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aac9f6cdfb84625b5b3438ae6e18d75" candidate_uuid: "1e513437eeac457aa8354f95edcbd9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "225f81a50ceb4492b32d74188461b59f" is_pre_election: true
I20260620 20:25:56.707320 15894 raft_consensus.cc:2379] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1e513437eeac457aa8354f95edcbd9ec in current term 1: Already voted for candidate f4de56503aca4b3096531ba2ea1a30d5 in this term.
I20260620 20:25:56.707460 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:56.698205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:36079 (local address 127.31.235.130:46717)
0620 20:25:56.698959 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 20:25:56.698979 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:25:56.699117 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:25:56.699468 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:25:56.699473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:25:56.699496 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:25:56.699830 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:56.699840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.701267 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.701274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:56.703638 (+ 2364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:25:56.703651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.705368 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.705377 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:25:56.705796 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:56.705897 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 20:25:56.706916 (+ 1019us) client_negotiation.cc:240] Negotiation successful
0620 20:25:56.707098 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":26,"thread_start_us":435,"threads_started":1}
I20260620 20:25:56.707477 16237 consensus_queue.cc:237] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } }
I20260620 20:25:56.708345 16162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aac9f6cdfb84625b5b3438ae6e18d75" candidate_uuid: "1e513437eeac457aa8354f95edcbd9ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4de56503aca4b3096531ba2ea1a30d5" is_pre_election: true
I20260620 20:25:56.709364 15961 leader_election.cc:302] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1e513437eeac457aa8354f95edcbd9ec; no voters: 225f81a50ceb4492b32d74188461b59f, f4de56503aca4b3096531ba2ea1a30d5
I20260620 20:25:56.709766 16232 raft_consensus.cc:2726] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 20:25:56.712442 15749 catalog_manager.cc:5495] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4de56503aca4b3096531ba2ea1a30d5 (127.31.235.131). New cstate: current_term: 1 leader_uuid: "f4de56503aca4b3096531ba2ea1a30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } health_report { overall_health: UNKNOWN } } }
I20260620 20:25:56.746719 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:56.741497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47200 (local address 127.31.235.131:36079)
0620 20:25:56.741736 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:56.741740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:56.741759 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:25:56.741954 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:56.741958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:56.742038 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:56.742161 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:25:56.742484 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.743251 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.744565 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.744954 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.745061 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:56.745559 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:25:56.746086 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:25:56.746201 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:56.746308 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 20:25:56.746373 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:25:56.754594 15894 raft_consensus.cc:1261] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Refusing update from remote peer f4de56503aca4b3096531ba2ea1a30d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:25:56.754594 16028 raft_consensus.cc:1261] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec [term 1 FOLLOWER]: Refusing update from remote peer f4de56503aca4b3096531ba2ea1a30d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:25:56.755437 16238 consensus_queue.cc:1035] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:25:56.755834 16237 consensus_queue.cc:1035] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:25:56.767733 16242 mvcc.cc:205] Tried to move back new op lower bound from 7299019394057445376 to 7299019393879379968. Current Snapshot: MvccSnapshot[applied={T|T < 7299019394057445376}]
I20260620 20:25:56.778321 15750 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:25:56.778596 15750 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:25:56.782035 15750 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6: Sending DeleteTablet for 3 replicas of tablet 2aac9f6cdfb84625b5b3438ae6e18d75
I20260620 20:25:56.783217 15874 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aac9f6cdfb84625b5b3438ae6e18d75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:25:56 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:40642
I20260620 20:25:56.783389 16142 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aac9f6cdfb84625b5b3438ae6e18d75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:25:56 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:47518
I20260620 20:25:56.783399 16008 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aac9f6cdfb84625b5b3438ae6e18d75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:25:56 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:46854
I20260620 20:25:56.783535 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 15811
I20260620 20:25:56.796372 16250 tablet_replica.cc:330] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f: stopping tablet replica
I20260620 20:25:56.796983 16250 raft_consensus.cc:2229] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:25:56.797645 16250 raft_consensus.cc:2258] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:25:56.800073 16250 ts_tablet_manager.cc:1905] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:25:56.806576 16251 tablet_replica.cc:330] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec: stopping tablet replica
I20260620 20:25:56.807193 16251 raft_consensus.cc:2229] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec [term 1 FOLLOWER]: Raft consensus shutting down.
Jun 20 20:25:56 dist-test-slave-zzwf krb5kdc[15705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:25:56.807839 16251 raft_consensus.cc:2258] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:25:56.808240 16250 ts_tablet_manager.cc:1918] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:25:56.808470 16250 log.cc:1196] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/2aac9f6cdfb84625b5b3438ae6e18d75
I20260620 20:25:56.808735 16252 tablet_replica.cc:330] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5: stopping tablet replica
I20260620 20:25:56.809084 16250 ts_tablet_manager.cc:1939] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 225f81a50ceb4492b32d74188461b59f: Deleting consensus metadata
I20260620 20:25:56.809166 16252 raft_consensus.cc:2229] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:25:56.809878 16252 raft_consensus.cc:2258] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:25:56.811998 15734 catalog_manager.cc:4841] TS 225f81a50ceb4492b32d74188461b59f (127.31.235.129:45307): tablet 2aac9f6cdfb84625b5b3438ae6e18d75 (table test-table [id=c20e29af234d4ce99604f2f26c32618d]) successfully deleted
I20260620 20:25:56.812381 16251 ts_tablet_manager.cc:1905] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:25:56.814886 16252 ts_tablet_manager.cc:1905] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:25:56.817173 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:56.784433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54988 (local address 127.31.235.129:45307)
0620 20:25:56.784655 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:56.784664 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:56.796091 (+ 11427us) server_negotiation.cc:407] Connection header received
0620 20:25:56.796482 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:56.796490 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:56.796582 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:56.796655 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:56.798999 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.800068 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.802281 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.802960 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.803056 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:56.809436 (+ 6380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:56.809450 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:56.809468 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:56.809534 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:56.813573 (+ 4039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:56.814513 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:56.814522 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:56.814526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:56.814684 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:56.815183 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:56.815188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:56.815191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:56.815666 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:56.815978 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:56.816571 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 20:25:56.816819 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:25:56.818615 15934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:56.822479 16252 ts_tablet_manager.cc:1918] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:25:56.822656 16252 log.cc:1196] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/2aac9f6cdfb84625b5b3438ae6e18d75
I20260620 20:25:56.822899 16251 ts_tablet_manager.cc:1918] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:25:56.823101 16251 log.cc:1196] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/2aac9f6cdfb84625b5b3438ae6e18d75
I20260620 20:25:56.823232 16252 ts_tablet_manager.cc:1939] T 2aac9f6cdfb84625b5b3438ae6e18d75 P f4de56503aca4b3096531ba2ea1a30d5: Deleting consensus metadata
I20260620 20:25:56.823695 16251 ts_tablet_manager.cc:1939] T 2aac9f6cdfb84625b5b3438ae6e18d75 P 1e513437eeac457aa8354f95edcbd9ec: Deleting consensus metadata
I20260620 20:25:56.825124 15734 catalog_manager.cc:4841] TS f4de56503aca4b3096531ba2ea1a30d5 (127.31.235.131:36079): tablet 2aac9f6cdfb84625b5b3438ae6e18d75 (table test-table [id=c20e29af234d4ce99604f2f26c32618d]) successfully deleted
I20260620 20:25:56.825438 15734 catalog_manager.cc:4841] TS 1e513437eeac457aa8354f95edcbd9ec (127.31.235.130:36427): tablet 2aac9f6cdfb84625b5b3438ae6e18d75 (table test-table [id=c20e29af234d4ce99604f2f26c32618d]) successfully deleted
I20260620 20:25:56.913470 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 15811
I20260620 20:25:56.938215 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 15945
Jun 20 20:25:56 dist-test-slave-zzwf krb5kdc[15705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.130@KRBTEST.COM
I20260620 20:25:56.953826 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:56.938960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33216 (local address 127.31.235.130:36427)
0620 20:25:56.939210 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:56.939215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:56.939235 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:25:56.939395 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:56.939401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:56.939482 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:56.939588 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:56.940810 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.941573 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.942864 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:56.943264 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:56.943398 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:56.948015 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:56.948029 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:56.948043 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:56.948077 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:56.950793 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:56.951605 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:56.951612 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:56.951614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:56.951730 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:56.952159 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:56.952164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:56.952166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:56.952583 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:56.952820 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:56.953189 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 20:25:56.953370 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260620 20:25:56.954523 16068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:57.050992 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 15945
I20260620 20:25:57.068764 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 16079
Jun 20 20:25:57 dist-test-slave-zzwf krb5kdc[15705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.131@KRBTEST.COM
I20260620 20:25:57.092002 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:25:57.069623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47202 (local address 127.31.235.131:36079)
0620 20:25:57.069868 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 20:25:57.069876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:25:57.069930 (+ 54us) server_negotiation.cc:407] Connection header received
0620 20:25:57.070131 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:25:57.070137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:25:57.070234 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:25:57.070358 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:25:57.071692 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:57.072858 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:57.074302 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:25:57.075023 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:25:57.075173 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:25:57.084252 (+ 9079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:25:57.084268 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:25:57.084288 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:25:57.084340 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:25:57.088480 (+ 4140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:57.089380 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:57.089389 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:57.089394 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:57.089605 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:25:57.090016 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:25:57.090021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:25:57.090024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:25:57.090557 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:25:57.090866 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 20:25:57.091206 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 20:25:57.091460 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 20:25:57.092869 16202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:57.215826 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 16079
I20260620 20:25:57.235879 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 15718
I20260620 20:25:57.237049 15780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:25:57.357959 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 15718
I20260620 20:26:17.380017 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:39161
--webserver_interface=127.31.235.190
--webserver_port=43605
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:39161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:26:17.708904 16258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:26:17.709208 16258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:26:17.709298 16258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:26:17.716737 16258 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 20:26:17.716831 16258 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 20:26:17.716871 16258 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 20:26:17.716917 16258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:26:17.716970 16258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:26:17.717044 16258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:26:17.717089 16258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:26:17.717149 16258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:26:17.727231 16258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:39161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:39161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=43605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:26:17.728457 16258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:26:17.729761 16258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:26:17.753561 16265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:26:17.753504 16267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:26:17.753584 16264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:26:17.754338 16258 server_base.cc:1034] running on GCE node
I20260620 20:26:17.755177 16258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:26:17.756649 16258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:26:17.757880 16258 hybrid_clock.cc:648] HybridClock initialized: now 1781987177757801 us; error 52 us; skew 500 ppm
I20260620 20:26:17.758266 16258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:26:17 dist-test-slave-zzwf krb5kdc[15705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987177, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:26:17.768918 16258 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:26:17.770813 16258 webserver.cc:460] Webserver started at http://127.31.235.190:43605/ using document root <none> and password file <none>
I20260620 20:26:17.771636 16258 fs_manager.cc:359] Metadata directory not provided
I20260620 20:26:17.771826 16258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:26:17.778225 16258 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 20:26:17.781406 16274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:17.782953 16258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:26:17.783151 16258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "37f76da362154684a3f612ac76e371b6"
format_stamp: "Formatted at 2026-06-20 20:25:54 on dist-test-slave-zzwf"
server_key: "86f2f19ac948bf51a8a7c5ed711070d3"
server_key_iv: "c0daea5a53a7ec262d3abacbd87ab597"
server_key_version: "encryptionkey@0"
I20260620 20:26:17.784212 16258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:26:17.796375 16258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:26:17.801151 16258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:26:17.801476 16258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:26:17.820695 16258 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:39161
I20260620 20:26:17.820735 16325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:39161 every 8 connection(s)
I20260620 20:26:17.822357 16258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:26:17.824020 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 16258
I20260620 20:26:17.824887 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:45307
--local_ip_for_outbound_sockets=127.31.235.129
--tserver_master_addrs=127.31.235.190:39161
--webserver_port=35663
--webserver_interface=127.31.235.129
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:26:17.830636 16326 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:26:17.834476 16326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6: Bootstrap starting.
I20260620 20:26:17.856851 16326 log.cc:824] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6: Log is configured to *not* fsync() on all Append() calls
I20260620 20:26:17.869823 16326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:26:17.870420 16326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6: Bootstrap complete.
I20260620 20:26:17.880524 16326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f76da362154684a3f612ac76e371b6" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39161 } }
I20260620 20:26:17.881466 16326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37f76da362154684a3f612ac76e371b6, State: Initialized, Role: FOLLOWER
I20260620 20:26:17.882128 16326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f76da362154684a3f612ac76e371b6" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39161 } }
I20260620 20:26:17.882390 16326 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:26:17.882534 16326 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:26:17.882752 16326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 1 FOLLOWER]: Advancing to term 2
I20260620 20:26:17.886929 16326 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f76da362154684a3f612ac76e371b6" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39161 } }
I20260620 20:26:17.887567 16326 leader_election.cc:302] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37f76da362154684a3f612ac76e371b6; no voters:
I20260620 20:26:17.888319 16326 leader_election.cc:288] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 20:26:17.888473 16330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 20:26:17.889675 16330 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 2 LEADER]: Becoming Leader. State: Replica: 37f76da362154684a3f612ac76e371b6, State: Running, Role: LEADER
I20260620 20:26:17.890312 16330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f76da362154684a3f612ac76e371b6" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39161 } }
I20260620 20:26:17.890851 16326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:26:17.892771 16332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37f76da362154684a3f612ac76e371b6. Latest consensus state: current_term: 2 leader_uuid: "37f76da362154684a3f612ac76e371b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f76da362154684a3f612ac76e371b6" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39161 } } }
I20260620 20:26:17.893162 16332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [sys.catalog]: This master's current role is: LEADER
I20260620 20:26:17.893728 16331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "37f76da362154684a3f612ac76e371b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f76da362154684a3f612ac76e371b6" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39161 } } }
I20260620 20:26:17.894033 16331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [sys.catalog]: This master's current role is: LEADER
I20260620 20:26:17.901844 16335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:26:17.918519 16335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:26:17.920149 16335 catalog_manager.cc:1259] Loaded cluster ID: a44ca722a80f40aabbcf41b0c8ad76d7
I20260620 20:26:17.920353 16335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:26:17.924440 16335 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:26:17.925376 16347 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 20:26:17.926158 16335 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6: Loaded TSK: 0
I20260620 20:26:17.963207 16335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6: Generated new TSK 1
I20260620 20:26:17.963615 16335 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 20:26:18.182364 16328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:26:18.182657 16328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:26:18.182808 16328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:26:18.190505 16328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:26:18.190629 16328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:26:18.190889 16328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:26:18.201244 16328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:45307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=35663
--tserver_master_addrs=127.31.235.190:39161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:26:18.202581 16328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:26:18.204057 16328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:26:18.219674 16354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:26:18.219718 16356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:26:18.222254 16353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:26:18.226353 16328 server_base.cc:1034] running on GCE node
I20260620 20:26:18.227120 16328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:26:18.228221 16328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:26:18.229563 16328 hybrid_clock.cc:648] HybridClock initialized: now 1781987178229432 us; error 63 us; skew 500 ppm
I20260620 20:26:18.230015 16328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:26:18 dist-test-slave-zzwf krb5kdc[15705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987178, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:26:18.233952 16328 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:26:18.235739 16328 webserver.cc:460] Webserver started at http://127.31.235.129:35663/ using document root <none> and password file <none>
I20260620 20:26:18.236519 16328 fs_manager.cc:359] Metadata directory not provided
I20260620 20:26:18.236687 16328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:26:18.242957 16328 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 20:26:18.245855 16363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:18.247332 16328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 20:26:18.247505 16328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "225f81a50ceb4492b32d74188461b59f"
format_stamp: "Formatted at 2026-06-20 20:25:55 on dist-test-slave-zzwf"
server_key: "4488c6bbdda90d61cd5a939548c80705"
server_key_iv: "8b1e017e2e26aca057ad3eea9628f09a"
server_key_version: "encryptionkey@0"
I20260620 20:26:18.249100 16328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:26:18.283548 16328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:26:18.287955 16328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:26:18.288232 16328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:26:18.289353 16328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:26:18.290761 16328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:26:18.290896 16328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:18.291057 16328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:26:18.291160 16328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:18.325258 16328 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:45307
I20260620 20:26:18.325369 16475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:45307 every 8 connection(s)
I20260620 20:26:18.326725 16328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:26:18.330513 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 16328
I20260620 20:26:18.331197 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:36427
--local_ip_for_outbound_sockets=127.31.235.130
--tserver_master_addrs=127.31.235.190:39161
--webserver_port=43767
--webserver_interface=127.31.235.130
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:26:18 dist-test-slave-zzwf krb5kdc[15705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987178, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:26:18.352399 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:18.329477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:56239 (local address 127.31.235.190:39161)
0620 20:26:18.330108 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:18.330127 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:18.330773 (+ 646us) server_negotiation.cc:407] Connection header received
0620 20:26:18.333003 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:18.333016 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:18.333343 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:18.333765 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:26:18.335288 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:18.336545 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:18.339055 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:18.339731 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:18.339871 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:18.344517 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:26:18.344526 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:26:18.344541 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:26:18.344587 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:26:18.347659 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:18.348685 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:18.348692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:18.348698 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:18.348847 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:18.349404 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:18.349407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:18.349409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:18.349937 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:26:18.350083 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 20:26:18.351126 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0620 20:26:18.351472 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":259,"threads_started":1}
I20260620 20:26:18.353250 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:18.329646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:56239)
0620 20:26:18.330545 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:18.330614 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 20:26:18.332456 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:26:18.333893 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:26:18.333904 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:26:18.334537 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:26:18.335073 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:18.335091 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:18.336740 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:18.336745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:18.338862 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:18.338881 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:18.339980 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:18.339989 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:18.340344 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:18.341109 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:26:18.341150 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:26:18.344221 (+ 3071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:26:18.347817 (+ 3596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:26:18.347830 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:26:18.347849 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:26:18.348542 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:26:18.348979 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:26:18.348987 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:26:18.348992 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:26:18.349276 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:26:18.350085 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:26:18.350097 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:26:18.350589 (+ 492us) client_negotiation.cc:769] Sending connection context
0620 20:26:18.350935 (+ 346us) client_negotiation.cc:240] Negotiation successful
0620 20:26:18.351590 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":387,"threads_started":1}
I20260620 20:26:18.356875 16476 heartbeater.cc:347] Connected to a master server at 127.31.235.190:39161
I20260620 20:26:18.357324 16476 heartbeater.cc:464] Registering TS with master...
I20260620 20:26:18.358237 16476 heartbeater.cc:511] Master 127.31.235.190:39161 requested a full tablet report, sending...
I20260620 20:26:18.360618 16291 ts_manager.cc:194] Registered new tserver with Master: 225f81a50ceb4492b32d74188461b59f (127.31.235.129:45307)
I20260620 20:26:18.362146 16291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:56239
W20260620 20:26:18.668007 16480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:26:18.668314 16480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:26:18.668476 16480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:26:18.676553 16480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:26:18.676676 16480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:26:18.676891 16480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:26:18.686883 16480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:36427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=43767
--tserver_master_addrs=127.31.235.190:39161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:26:18.688417 16480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:26:18.689931 16480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:26:18.703474 16485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:26:18.703399 16486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:26:18.703444 16488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:26:18.704499 16480 server_base.cc:1034] running on GCE node
I20260620 20:26:18.705144 16480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:26:18.706182 16480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:26:18.707434 16480 hybrid_clock.cc:648] HybridClock initialized: now 1781987178707396 us; error 50 us; skew 500 ppm
I20260620 20:26:18.707865 16480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:26:18 dist-test-slave-zzwf krb5kdc[15705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987178, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:26:18.711165 16480 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:26:18.712677 16480 webserver.cc:460] Webserver started at http://127.31.235.130:43767/ using document root <none> and password file <none>
I20260620 20:26:18.713382 16480 fs_manager.cc:359] Metadata directory not provided
I20260620 20:26:18.713564 16480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:26:18.718987 16480 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:26:18.721937 16495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:18.723318 16480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:26:18.723507 16480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "1e513437eeac457aa8354f95edcbd9ec"
format_stamp: "Formatted at 2026-06-20 20:25:55 on dist-test-slave-zzwf"
server_key: "fb5e88801ae36d4fe06b819cf2966509"
server_key_iv: "0d924090dcd9c20b8de5b97fce19af5b"
server_key_version: "encryptionkey@0"
I20260620 20:26:18.724495 16480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:26:18.775172 16480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:26:18.779407 16480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:26:18.779699 16480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:26:18.780714 16480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:26:18.782368 16480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:26:18.782500 16480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:18.782642 16480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:26:18.782733 16480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:18.817112 16480 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:36427
I20260620 20:26:18.817205 16607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:36427 every 8 connection(s)
I20260620 20:26:18.818646 16480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:26:18.827831 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 16480
I20260620 20:26:18.828532 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:36079
--local_ip_for_outbound_sockets=127.31.235.131
--tserver_master_addrs=127.31.235.190:39161
--webserver_port=46867
--webserver_interface=127.31.235.131
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 20:26:18 dist-test-slave-zzwf krb5kdc[15705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987178, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:26:18.847024 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:18.821472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:57799 (local address 127.31.235.190:39161)
0620 20:26:18.821734 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:18.821740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:18.822981 (+ 1241us) server_negotiation.cc:407] Connection header received
0620 20:26:18.824846 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:18.824852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:18.824968 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:18.825079 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:26:18.826559 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:18.827292 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:18.829309 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:18.829931 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:18.830060 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:18.840087 (+ 10027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:26:18.840095 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:26:18.840100 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:26:18.840151 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:26:18.843480 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:18.844459 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:18.844464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:18.844466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:18.844587 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:18.845168 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:18.845172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:18.845174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:18.845585 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:26:18.845713 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 20:26:18.846593 (+ 880us) server_negotiation.cc:299] Negotiation successful
0620 20:26:18.846774 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260620 20:26:18.848567 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:18.822014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.130:57799)
0620 20:26:18.822825 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:18.822873 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:26:18.824580 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:26:18.825309 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:26:18.825316 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:26:18.825886 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:26:18.826397 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:18.826413 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:18.827480 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:18.827484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:18.829158 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:18.829170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:18.830158 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:18.830167 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:18.830411 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:18.837199 (+ 6788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:26:18.837224 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:26:18.839788 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:26:18.843637 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:26:18.843648 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:26:18.843669 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:26:18.844309 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:26:18.844709 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:26:18.844717 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:26:18.844724 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:26:18.844978 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:26:18.845719 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:26:18.845729 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:26:18.846199 (+ 470us) client_negotiation.cc:769] Sending connection context
0620 20:26:18.846500 (+ 301us) client_negotiation.cc:240] Negotiation successful
0620 20:26:18.847090 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":340,"threads_started":1}
I20260620 20:26:18.850153 16608 heartbeater.cc:347] Connected to a master server at 127.31.235.190:39161
I20260620 20:26:18.850474 16608 heartbeater.cc:464] Registering TS with master...
I20260620 20:26:18.851189 16608 heartbeater.cc:511] Master 127.31.235.190:39161 requested a full tablet report, sending...
I20260620 20:26:18.852839 16291 ts_manager.cc:194] Registered new tserver with Master: 1e513437eeac457aa8354f95edcbd9ec (127.31.235.130:36427)
I20260620 20:26:18.853816 16291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:57799
W20260620 20:26:19.161751 16611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:26:19.162017 16611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:26:19.162110 16611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:26:19.169234 16611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:26:19.169338 16611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:26:19.169447 16611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:26:19.179567 16611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:36079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=46867
--tserver_master_addrs=127.31.235.190:39161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:26:19.180924 16611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:26:19.182293 16611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:26:19.195860 16616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:26:19.195921 16617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:26:19.195983 16619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:26:19.196594 16611 server_base.cc:1034] running on GCE node
I20260620 20:26:19.197283 16611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:26:19.198359 16611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:26:19.199591 16611 hybrid_clock.cc:648] HybridClock initialized: now 1781987179199532 us; error 56 us; skew 500 ppm
I20260620 20:26:19.199931 16611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:26:19 dist-test-slave-zzwf krb5kdc[15705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987179, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:26:19.203128 16611 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:26:19.204700 16611 webserver.cc:460] Webserver started at http://127.31.235.131:46867/ using document root <none> and password file <none>
I20260620 20:26:19.205386 16611 fs_manager.cc:359] Metadata directory not provided
I20260620 20:26:19.205562 16611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:26:19.212174 16611 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:26:19.215245 16626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:19.216647 16611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:26:19.216823 16611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "f4de56503aca4b3096531ba2ea1a30d5"
format_stamp: "Formatted at 2026-06-20 20:25:56 on dist-test-slave-zzwf"
server_key: "2a794319c1cdb4011cca3c54091d52cf"
server_key_iv: "7ca62692cd477ae84f3f61e22ec8946d"
server_key_version: "encryptionkey@0"
I20260620 20:26:19.217862 16611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:26:19.233178 16611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:26:19.237414 16611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:26:19.237730 16611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:26:19.238744 16611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:26:19.240134 16611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:26:19.240257 16611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:19.240391 16611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:26:19.240480 16611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:19.274718 16611 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:36079
I20260620 20:26:19.274807 16738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:36079 every 8 connection(s)
I20260620 20:26:19.276223 16611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:26:19.277680 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 16611
Jun 20 20:26:19 dist-test-slave-zzwf krb5kdc[15705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987179, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:26:19.296165 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.278710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52406 (local address 127.31.235.190:39161)
0620 20:26:19.278917 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:19.278925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:19.281777 (+ 2852us) server_negotiation.cc:407] Connection header received
0620 20:26:19.281978 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:19.281982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:19.282049 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:19.282161 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:26:19.283660 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.284408 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.285951 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.286364 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.286495 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:19.289617 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:26:19.289622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:26:19.289626 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:26:19.289656 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:26:19.293226 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:19.293952 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:19.293959 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:19.293963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:19.294134 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:19.294552 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:19.294557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:19.294560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:19.294965 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:26:19.295264 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 20:26:19.295564 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 20:26:19.295783 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"spinlock_wait_cycles":3840}
I20260620 20:26:19.298990 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.278836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:42493 (local address 127.31.235.190:39161)
0620 20:26:19.279270 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:19.279273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:19.280667 (+ 1394us) server_negotiation.cc:407] Connection header received
0620 20:26:19.282729 (+ 2062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:19.282734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:19.282809 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:19.282940 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:26:19.284386 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.285400 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.287013 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.287534 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.287668 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:19.291204 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:26:19.291209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:26:19.291212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:26:19.291273 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:26:19.295554 (+ 4281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:19.296412 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:19.296417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:19.296420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:19.296546 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:19.297051 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:19.297058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:19.297061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:19.297407 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:26:19.297578 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 20:26:19.298304 (+ 726us) server_negotiation.cc:299] Negotiation successful
0620 20:26:19.298561 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":258,"threads_started":1}
I20260620 20:26:19.299785 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.279695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.131:42493)
0620 20:26:19.280500 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:19.280541 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 20:26:19.282476 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:26:19.283175 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:26:19.283184 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:26:19.283659 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:26:19.284196 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:19.284211 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.285596 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.285601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:19.286867 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:19.286879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.287810 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.287821 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:19.288030 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:19.288700 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:26:19.288721 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:26:19.290968 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:26:19.295722 (+ 4754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:26:19.295732 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:26:19.295750 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:26:19.296277 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:26:19.296684 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:26:19.296692 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:26:19.296697 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:26:19.296932 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:26:19.297589 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:26:19.297599 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:26:19.297966 (+ 367us) client_negotiation.cc:769] Sending connection context
0620 20:26:19.298204 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 20:26:19.298707 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":348,"threads_started":1}
I20260620 20:26:19.301105 16739 heartbeater.cc:347] Connected to a master server at 127.31.235.190:39161
I20260620 20:26:19.301441 16739 heartbeater.cc:464] Registering TS with master...
I20260620 20:26:19.302165 16739 heartbeater.cc:511] Master 127.31.235.190:39161 requested a full tablet report, sending...
I20260620 20:26:19.303547 16291 ts_manager.cc:194] Registered new tserver with Master: f4de56503aca4b3096531ba2ea1a30d5 (127.31.235.131:36079)
I20260620 20:26:19.304494 16291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:42493
I20260620 20:26:19.311314 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:26:19.318171 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.312384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52408 (local address 127.31.235.190:39161)
0620 20:26:19.312545 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:19.312549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:19.313017 (+ 468us) server_negotiation.cc:407] Connection header received
0620 20:26:19.313221 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:19.313224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:19.313399 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:19.313612 (+ 213us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:26:19.314028 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.314763 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.316350 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.316767 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.316861 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:19.317417 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:26:19.317699 (+ 282us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 20:26:19.317979 (+ 280us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52408: token expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 20:26:19.318322 16742 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52408: token expired
W20260620 20:26:19.319068 16213 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.31.235.190:39161)
I20260620 20:26:19.332463 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.319655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52418 (local address 127.31.235.190:39161)
0620 20:26:19.319807 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:19.319811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:19.319826 (+ 15us) server_negotiation.cc:407] Connection header received
0620 20:26:19.319895 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:19.319898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:19.319947 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:19.320019 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:26:19.321428 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.322138 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.323465 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.323833 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.323921 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:19.326730 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:26:19.326738 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:26:19.326748 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:26:19.326781 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:26:19.329767 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:19.330431 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:19.330437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:19.330441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:19.330581 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:19.331034 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:19.331040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:19.331043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:19.331350 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:26:19.331619 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 20:26:19.331933 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 20:26:19.332141 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:26:19.335922 16291 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:26:19.343881 16291 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:26:19.365734 16476 heartbeater.cc:503] Master 127.31.235.190:39161 was elected leader, sending a full tablet report...
I20260620 20:26:19.369977 16749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.356325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:45307 (local address 127.0.0.1:60446)
0620 20:26:19.359892 (+ 3567us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:19.359935 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:26:19.360138 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:26:19.361634 (+ 1496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:26:19.361643 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:26:19.361677 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:26:19.362142 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:19.362159 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.364172 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.364177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:19.366957 (+ 2780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:19.366972 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.368885 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.368892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:19.369102 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:19.369163 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 20:26:19.369294 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 20:26:19.369514 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3263,"mutex_wait_us":44,"thread_start_us":3107,"threads_started":1}
I20260620 20:26:19.370005 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.356614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.31.235.129:45307)
0620 20:26:19.360750 (+ 4136us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:19.360759 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:19.360791 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:26:19.360890 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:19.360897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:19.361197 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:19.361447 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:26:19.362369 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.363993 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.367144 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.368673 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.368885 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:19.369071 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 20:26:19.369387 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 20:26:19.369633 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3940,"thread_start_us":3741,"threads_started":1}
I20260620 20:26:19.370523 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.360510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:36427 (local address 127.0.0.1:35462)
0620 20:26:19.361110 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:19.361128 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:26:19.361246 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:26:19.362658 (+ 1412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:26:19.362663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:26:19.362690 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:26:19.363137 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:19.363150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.365061 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.365066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:19.367295 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:19.367311 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.369636 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.369642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:19.369833 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:19.369876 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 20:26:19.369946 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 20:26:19.370093 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":20,"thread_start_us":332,"threads_started":1}
I20260620 20:26:19.371600 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.360833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35462 (local address 127.31.235.130:36427)
0620 20:26:19.362115 (+ 1282us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:19.362121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:19.362143 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:26:19.362210 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:19.362215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:19.362446 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:19.363413 (+ 967us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:26:19.363631 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.364924 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.367972 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.369365 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.370862 (+ 1497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:19.371021 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 20:26:19.371085 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 20:26:19.371244 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1130,"thread_start_us":1030,"threads_started":1}
I20260620 20:26:19.371805 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.362824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:36079 (local address 127.0.0.1:49380)
0620 20:26:19.363433 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:19.363452 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:26:19.363636 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:26:19.364438 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:26:19.364442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:26:19.364466 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:26:19.364848 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:19.364862 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.366644 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.366649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:19.368547 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:19.368559 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.371027 (+ 2468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.371031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:19.371141 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:19.371162 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:26:19.371267 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 20:26:19.371394 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":328,"threads_started":1}
I20260620 20:26:19.371958 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.361564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49380 (local address 127.31.235.131:36079)
0620 20:26:19.362803 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:19.362810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:19.363537 (+ 727us) server_negotiation.cc:407] Connection header received
0620 20:26:19.363834 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:19.363842 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:19.364039 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:19.364195 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:26:19.365050 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.366472 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.368690 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.370492 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.370800 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:19.370916 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 20:26:19.371339 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 20:26:19.371499 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1096,"thread_start_us":932,"threads_started":1}
I20260620 20:26:19.374224 16411 tablet_service.cc:1467] Processing CreateTablet for tablet da81de0ad9df40d9bb6f594306a31f74 (DEFAULT_TABLE table=test-table [id=61d17d9cc6304ab5b895fe9abe9b297c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:26:19.375512 16411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da81de0ad9df40d9bb6f594306a31f74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:26:19.375301 16674 tablet_service.cc:1467] Processing CreateTablet for tablet da81de0ad9df40d9bb6f594306a31f74 (DEFAULT_TABLE table=test-table [id=61d17d9cc6304ab5b895fe9abe9b297c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:26:19.376451 16674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da81de0ad9df40d9bb6f594306a31f74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:26:19.378209 16543 tablet_service.cc:1467] Processing CreateTablet for tablet da81de0ad9df40d9bb6f594306a31f74 (DEFAULT_TABLE table=test-table [id=61d17d9cc6304ab5b895fe9abe9b297c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:26:19.379633 16543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da81de0ad9df40d9bb6f594306a31f74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:26:19.389387 16755 tablet_bootstrap.cc:492] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5: Bootstrap starting.
I20260620 20:26:19.393029 16755 tablet_bootstrap.cc:654] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5: Neither blocks nor log segments found. Creating new log.
I20260620 20:26:19.393324 16756 tablet_bootstrap.cc:492] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f: Bootstrap starting.
I20260620 20:26:19.394587 16755 log.cc:824] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5: Log is configured to *not* fsync() on all Append() calls
I20260620 20:26:19.397228 16756 tablet_bootstrap.cc:654] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f: Neither blocks nor log segments found. Creating new log.
I20260620 20:26:19.398943 16756 log.cc:824] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:26:19.401302 16757 tablet_bootstrap.cc:492] T da81de0ad9df40d9bb6f594306a31f74 P 1e513437eeac457aa8354f95edcbd9ec: Bootstrap starting.
I20260620 20:26:19.402309 16755 tablet_bootstrap.cc:492] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5: No bootstrap required, opened a new log
I20260620 20:26:19.402624 16755 ts_tablet_manager.cc:1397] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260620 20:26:19.402884 16756 tablet_bootstrap.cc:492] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f: No bootstrap required, opened a new log
I20260620 20:26:19.403160 16756 ts_tablet_manager.cc:1397] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 20:26:19.404671 16757 tablet_bootstrap.cc:654] T da81de0ad9df40d9bb6f594306a31f74 P 1e513437eeac457aa8354f95edcbd9ec: Neither blocks nor log segments found. Creating new log.
I20260620 20:26:19.405781 16757 log.cc:824] T da81de0ad9df40d9bb6f594306a31f74 P 1e513437eeac457aa8354f95edcbd9ec: Log is configured to *not* fsync() on all Append() calls
I20260620 20:26:19.411726 16757 tablet_bootstrap.cc:492] T da81de0ad9df40d9bb6f594306a31f74 P 1e513437eeac457aa8354f95edcbd9ec: No bootstrap required, opened a new log
I20260620 20:26:19.412029 16757 ts_tablet_manager.cc:1397] T da81de0ad9df40d9bb6f594306a31f74 P 1e513437eeac457aa8354f95edcbd9ec: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260620 20:26:19.413998 16755 raft_consensus.cc:348] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } }
I20260620 20:26:19.414000 16756 raft_consensus.cc:348] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } }
I20260620 20:26:19.414443 16755 raft_consensus.cc:374] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:26:19.414441 16756 raft_consensus.cc:374] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:26:19.414549 16756 raft_consensus.cc:729] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 225f81a50ceb4492b32d74188461b59f, State: Initialized, Role: FOLLOWER
I20260620 20:26:19.414541 16755 raft_consensus.cc:729] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4de56503aca4b3096531ba2ea1a30d5, State: Initialized, Role: FOLLOWER
I20260620 20:26:19.415189 16755 consensus_queue.cc:260] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } }
I20260620 20:26:19.415189 16756 consensus_queue.cc:260] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } }
I20260620 20:26:19.417045 16755 ts_tablet_manager.cc:1428] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260620 20:26:19.417264 16756 ts_tablet_manager.cc:1428] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260620 20:26:19.417608 16739 heartbeater.cc:503] Master 127.31.235.190:39161 was elected leader, sending a full tablet report...
I20260620 20:26:19.421420 16757 raft_consensus.cc:348] T da81de0ad9df40d9bb6f594306a31f74 P 1e513437eeac457aa8354f95edcbd9ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } }
I20260620 20:26:19.421928 16757 raft_consensus.cc:374] T da81de0ad9df40d9bb6f594306a31f74 P 1e513437eeac457aa8354f95edcbd9ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:26:19.422035 16757 raft_consensus.cc:729] T da81de0ad9df40d9bb6f594306a31f74 P 1e513437eeac457aa8354f95edcbd9ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e513437eeac457aa8354f95edcbd9ec, State: Initialized, Role: FOLLOWER
I20260620 20:26:19.422776 16757 consensus_queue.cc:260] T da81de0ad9df40d9bb6f594306a31f74 P 1e513437eeac457aa8354f95edcbd9ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } }
I20260620 20:26:19.423920 16608 heartbeater.cc:503] Master 127.31.235.190:39161 was elected leader, sending a full tablet report...
I20260620 20:26:19.425773 16757 ts_tablet_manager.cc:1428] T da81de0ad9df40d9bb6f594306a31f74 P 1e513437eeac457aa8354f95edcbd9ec: Time spent starting tablet: real 0.014s user 0.006s sys 0.008s
I20260620 20:26:19.456562 16761 raft_consensus.cc:482] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:26:19.456905 16761 raft_consensus.cc:504] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } }
I20260620 20:26:19.458652 16761 leader_election.cc:288] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e513437eeac457aa8354f95edcbd9ec (127.31.235.130:36427), f4de56503aca4b3096531ba2ea1a30d5 (127.31.235.131:36079)
I20260620 20:26:19.465945 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.459059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:38223 (local address 127.31.235.130:36427)
0620 20:26:19.459300 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:19.459309 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:19.459975 (+ 666us) server_negotiation.cc:407] Connection header received
0620 20:26:19.460208 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:19.460214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:19.460293 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:19.460367 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:26:19.461191 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.462148 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.464050 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.464909 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.465026 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:19.465077 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 20:26:19.465532 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 20:26:19.465665 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:26:19.466272 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.460089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:45105 (local address 127.31.235.131:36079)
0620 20:26:19.460296 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:19.460302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:19.460616 (+ 314us) server_negotiation.cc:407] Connection header received
0620 20:26:19.460755 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:19.460760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:19.460835 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:19.460941 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:26:19.461657 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.462507 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.464323 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.465225 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.465383 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:19.465471 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 20:26:19.465842 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 20:26:19.465981 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 20:26:19.466289 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.459064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:36427 (local address 127.31.235.129:38223)
0620 20:26:19.459801 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:19.459854 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 20:26:19.460016 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:26:19.460536 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:26:19.460539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:26:19.460587 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:26:19.461013 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:19.461027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.462300 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.462305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:19.463883 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:19.463896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.465130 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.465136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:19.465326 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:19.465362 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 20:26:19.465466 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 20:26:19.465623 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":420,"threads_started":1}
I20260620 20:26:19.466356 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.459899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:36079 (local address 127.31.235.129:45105)
0620 20:26:19.460510 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:19.460524 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:26:19.460627 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:26:19.461003 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:26:19.461007 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:26:19.461029 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:26:19.461460 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:19.461473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.462663 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.462670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:19.464153 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:19.464168 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.465450 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.465458 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:19.465658 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:19.465689 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 20:26:19.465782 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 20:26:19.465900 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":303,"threads_started":1}
I20260620 20:26:19.467234 16563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da81de0ad9df40d9bb6f594306a31f74" candidate_uuid: "225f81a50ceb4492b32d74188461b59f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e513437eeac457aa8354f95edcbd9ec" is_pre_election: true
I20260620 20:26:19.467279 16694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da81de0ad9df40d9bb6f594306a31f74" candidate_uuid: "225f81a50ceb4492b32d74188461b59f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4de56503aca4b3096531ba2ea1a30d5" is_pre_election: true
I20260620 20:26:19.467739 16563 raft_consensus.cc:2454] T da81de0ad9df40d9bb6f594306a31f74 P 1e513437eeac457aa8354f95edcbd9ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 225f81a50ceb4492b32d74188461b59f in term 0.
I20260620 20:26:19.467739 16694 raft_consensus.cc:2454] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 225f81a50ceb4492b32d74188461b59f in term 0.
I20260620 20:26:19.468381 16364 leader_election.cc:302] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 225f81a50ceb4492b32d74188461b59f, f4de56503aca4b3096531ba2ea1a30d5; no voters:
I20260620 20:26:19.468762 16761 raft_consensus.cc:2781] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:26:19.468919 16761 raft_consensus.cc:482] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:26:19.469036 16761 raft_consensus.cc:3037] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:26:19.471294 16761 raft_consensus.cc:504] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } }
I20260620 20:26:19.471921 16761 leader_election.cc:288] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 1 election: Requested vote from peers 1e513437eeac457aa8354f95edcbd9ec (127.31.235.130:36427), f4de56503aca4b3096531ba2ea1a30d5 (127.31.235.131:36079)
I20260620 20:26:19.472446 16563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da81de0ad9df40d9bb6f594306a31f74" candidate_uuid: "225f81a50ceb4492b32d74188461b59f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e513437eeac457aa8354f95edcbd9ec"
I20260620 20:26:19.472503 16694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da81de0ad9df40d9bb6f594306a31f74" candidate_uuid: "225f81a50ceb4492b32d74188461b59f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4de56503aca4b3096531ba2ea1a30d5"
I20260620 20:26:19.472640 16563 raft_consensus.cc:3037] T da81de0ad9df40d9bb6f594306a31f74 P 1e513437eeac457aa8354f95edcbd9ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:26:19.472792 16694 raft_consensus.cc:3037] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:26:19.474990 16563 raft_consensus.cc:2454] T da81de0ad9df40d9bb6f594306a31f74 P 1e513437eeac457aa8354f95edcbd9ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 225f81a50ceb4492b32d74188461b59f in term 1.
I20260620 20:26:19.475206 16694 raft_consensus.cc:2454] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 225f81a50ceb4492b32d74188461b59f in term 1.
I20260620 20:26:19.475463 16364 leader_election.cc:302] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e513437eeac457aa8354f95edcbd9ec, 225f81a50ceb4492b32d74188461b59f; no voters:
I20260620 20:26:19.475768 16761 raft_consensus.cc:2781] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:26:19.476240 16761 raft_consensus.cc:686] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 1 LEADER]: Becoming Leader. State: Replica: 225f81a50ceb4492b32d74188461b59f, State: Running, Role: LEADER
I20260620 20:26:19.476722 16761 consensus_queue.cc:237] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } }
I20260620 20:26:19.481599 16291 catalog_manager.cc:5495] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f reported cstate change: term changed from 0 to 1, leader changed from <none> to 225f81a50ceb4492b32d74188461b59f (127.31.235.129). New cstate: current_term: 1 leader_uuid: "225f81a50ceb4492b32d74188461b59f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } health_report { overall_health: UNKNOWN } } }
I20260620 20:26:19.522143 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.516795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.31.235.129:45307)
0620 20:26:19.517024 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:19.517030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:19.517051 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:26:19.517196 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:19.517200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:19.517288 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:19.517440 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:26:19.517872 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.518578 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.519931 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.520323 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.520450 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:19.521001 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:26:19.521567 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:26:19.521741 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 20:26:19.521799 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 20:26:19.521882 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 20:26:19.527900 16740 tablet.cc:2369] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:26:19.530435 16694 raft_consensus.cc:1261] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5 [term 1 FOLLOWER]: Refusing update from remote peer 225f81a50ceb4492b32d74188461b59f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:26:19.530601 16563 raft_consensus.cc:1261] T da81de0ad9df40d9bb6f594306a31f74 P 1e513437eeac457aa8354f95edcbd9ec [term 1 FOLLOWER]: Refusing update from remote peer 225f81a50ceb4492b32d74188461b59f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:26:19.531468 16761 consensus_queue.cc:1035] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:26:19.531834 16766 consensus_queue.cc:1035] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:26:19.543506 16769 mvcc.cc:205] Tried to move back new op lower bound from 7299019487346323456 to 7299019487140732928. Current Snapshot: MvccSnapshot[applied={T|T < 7299019487346323456}]
I20260620 20:26:19.544174 16771 mvcc.cc:205] Tried to move back new op lower bound from 7299019487346323456 to 7299019487140732928. Current Snapshot: MvccSnapshot[applied={T|T < 7299019487346323456}]
I20260620 20:26:19.558799 16291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:26:19.559080 16291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:26:19.563041 16291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6: Sending DeleteTablet for 3 replicas of tablet da81de0ad9df40d9bb6f594306a31f74
I20260620 20:26:19.564363 16411 tablet_service.cc:1514] Processing DeleteTablet for tablet da81de0ad9df40d9bb6f594306a31f74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:26:19 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:60446
I20260620 20:26:19.564494 16674 tablet_service.cc:1514] Processing DeleteTablet for tablet da81de0ad9df40d9bb6f594306a31f74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:26:19 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:49380
I20260620 20:26:19.564600 16543 tablet_service.cc:1514] Processing DeleteTablet for tablet da81de0ad9df40d9bb6f594306a31f74 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:26:19 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:35462
I20260620 20:26:19.565055 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 16328
I20260620 20:26:19.582139 16781 tablet_replica.cc:330] T da81de0ad9df40d9bb6f594306a31f74 P 1e513437eeac457aa8354f95edcbd9ec: stopping tablet replica
I20260620 20:26:19.582837 16781 raft_consensus.cc:2229] T da81de0ad9df40d9bb6f594306a31f74 P 1e513437eeac457aa8354f95edcbd9ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:26:19.583608 16781 raft_consensus.cc:2258] T da81de0ad9df40d9bb6f594306a31f74 P 1e513437eeac457aa8354f95edcbd9ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:26:19.586510 16781 ts_tablet_manager.cc:1905] T da81de0ad9df40d9bb6f594306a31f74 P 1e513437eeac457aa8354f95edcbd9ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:26:19.595342 16780 tablet_replica.cc:330] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5: stopping tablet replica
I20260620 20:26:19.595590 16781 ts_tablet_manager.cc:1918] T da81de0ad9df40d9bb6f594306a31f74 P 1e513437eeac457aa8354f95edcbd9ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:26:19.595829 16781 log.cc:1196] T da81de0ad9df40d9bb6f594306a31f74 P 1e513437eeac457aa8354f95edcbd9ec: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/da81de0ad9df40d9bb6f594306a31f74
I20260620 20:26:19.595861 16780 raft_consensus.cc:2229] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:26:19.596295 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.566921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60474 (local address 127.31.235.129:45307)
0620 20:26:19.567136 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:19.567144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:19.567167 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:26:19.567242 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:19.567246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:19.567348 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:19.567474 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:26:19.570025 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.574324 (+ 4299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.577036 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.578072 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.578225 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:19.584680 (+ 6455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:26:19.584701 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:26:19.584725 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:26:19.584798 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:26:19.590398 (+ 5600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:19.591739 (+ 1341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:19.591766 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:19.591773 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:19.592082 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:19.592686 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:19.592695 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:19.592700 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:19.593317 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:26:19.593703 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0620 20:26:19.595378 (+ 1675us) server_negotiation.cc:299] Negotiation successful
0620 20:26:19.595718 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 20:26:19.596459 16780 raft_consensus.cc:2258] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:26:19.597077 16781 ts_tablet_manager.cc:1939] T da81de0ad9df40d9bb6f594306a31f74 P 1e513437eeac457aa8354f95edcbd9ec: Deleting consensus metadata
I20260620 20:26:19.597147 16471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:26:19.599192 16275 catalog_manager.cc:4841] TS 1e513437eeac457aa8354f95edcbd9ec (127.31.235.130:36427): tablet da81de0ad9df40d9bb6f594306a31f74 (table test-table [id=61d17d9cc6304ab5b895fe9abe9b297c]) successfully deleted
I20260620 20:26:19.599308 16780 ts_tablet_manager.cc:1905] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:26:19.605810 16780 ts_tablet_manager.cc:1918] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:26:19.606001 16780 log.cc:1196] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/da81de0ad9df40d9bb6f594306a31f74
I20260620 20:26:19.606427 16780 ts_tablet_manager.cc:1939] T da81de0ad9df40d9bb6f594306a31f74 P f4de56503aca4b3096531ba2ea1a30d5: Deleting consensus metadata
I20260620 20:26:19.607929 16275 catalog_manager.cc:4841] TS f4de56503aca4b3096531ba2ea1a30d5 (127.31.235.131:36079): tablet da81de0ad9df40d9bb6f594306a31f74 (table test-table [id=61d17d9cc6304ab5b895fe9abe9b297c]) successfully deleted
I20260620 20:26:19.694949 16779 tablet_replica.cc:330] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f: stopping tablet replica
I20260620 20:26:19.695608 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 16328
W20260620 20:26:19.713935 16275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.235.129:45307 (error 108)
I20260620 20:26:19.714633 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 16480
I20260620 20:26:19.715955 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.715379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:45307 (local address 127.0.0.1:60488)
0620 20:26:19.715623 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:19.715784 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:45307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260620 20:26:19.716207 16275 catalog_manager.cc:4584] TS 225f81a50ceb4492b32d74188461b59f (127.31.235.129:45307): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da81de0ad9df40d9bb6f594306a31f74: Network error: Client connection negotiation failed: client connection to 127.31.235.129:45307: connect: Connection refused (error 111)
I20260620 20:26:19.732210 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.715407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.31.235.130:36427)
0620 20:26:19.715579 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:19.715585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:19.715606 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:26:19.715789 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:19.715795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:19.715876 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:19.716000 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:26:19.717378 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.718226 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.719551 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.719945 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.720049 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:19.726420 (+ 6371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:26:19.726436 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:26:19.726452 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:26:19.726487 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:26:19.729446 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:19.730384 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:19.730391 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:19.730393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:19.730513 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:19.730903 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:19.730908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:19.730910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:19.731311 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:26:19.731545 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 20:26:19.731793 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 20:26:19.731958 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:26:19.732906 16603 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:26:19.769852 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.769262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:45307 (local address 127.0.0.1:60498)
0620 20:26:19.769529 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:19.769652 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:45307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 20:26:19.771015 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.770619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:45307 (local address 127.0.0.1:60502)
0620 20:26:19.770786 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:19.770873 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:45307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 20:26:19.827639 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 16480
I20260620 20:26:19.831604 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.831115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:45307 (local address 127.0.0.1:60514)
0620 20:26:19.831378 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:19.831457 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:45307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 20:26:19.832631 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.832305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:45307 (local address 127.0.0.1:60522)
0620 20:26:19.832446 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:19.832522 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:45307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 20:26:19.850150 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 16611
I20260620 20:26:19.864977 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.850944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49390 (local address 127.31.235.131:36079)
0620 20:26:19.851121 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:19.851125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:19.851144 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:26:19.851386 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:19.851391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:19.851479 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:19.851574 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:26:19.853210 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.854066 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.855583 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:19.856024 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:19.856128 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:19.859161 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:26:19.859175 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:26:19.859193 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:26:19.859236 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:26:19.862064 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:19.862841 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:19.862848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:19.862851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:19.862993 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:19.863387 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:19.863391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:19.863393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:19.863789 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:26:19.864016 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 20:26:19.864431 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 20:26:19.864694 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:26:19.865688 16734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:26:19.913033 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.912492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:45307 (local address 127.0.0.1:60530)
0620 20:26:19.912728 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:19.912842 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:45307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 20:26:19.914203 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:19.913763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:45307 (local address 127.0.0.1:60538)
0620 20:26:19.913977 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:19.914063 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:45307: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 20:26:19.956007 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 16611
I20260620 20:26:19.972965 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 16258
I20260620 20:26:19.974110 16321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:26:20.089316 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 16258
I20260620 20:26:40.111812 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:39161
--webserver_interface=127.31.235.190
--webserver_port=43605
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:39161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:26:40.453796 16787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:26:40.454111 16787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:26:40.454283 16787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:26:40.461608 16787 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 20:26:40.461727 16787 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 20:26:40.461815 16787 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 20:26:40.461915 16787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:26:40.461995 16787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:26:40.462096 16787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:26:40.462191 16787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:26:40.462272 16787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:26:40.475446 16787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:39161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:39161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=43605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:26:40.477197 16787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:26:40.478575 16787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:26:40.491044 16793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:26:40.491026 16795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:26:40.491106 16787 server_base.cc:1034] running on GCE node
W20260620 20:26:40.491029 16792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:26:40.492340 16787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:26:40.493882 16787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:26:40.495218 16787 hybrid_clock.cc:648] HybridClock initialized: now 1781987200495154 us; error 38 us; skew 500 ppm
I20260620 20:26:40.495649 16787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:26:40 dist-test-slave-zzwf krb5kdc[15705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987200, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:26:40.499069 16787 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:26:40.500690 16787 webserver.cc:460] Webserver started at http://127.31.235.190:43605/ using document root <none> and password file <none>
I20260620 20:26:40.501436 16787 fs_manager.cc:359] Metadata directory not provided
I20260620 20:26:40.501597 16787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:26:40.507326 16787 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 20:26:40.510370 16802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:40.511770 16787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:26:40.511952 16787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "37f76da362154684a3f612ac76e371b6"
format_stamp: "Formatted at 2026-06-20 20:25:54 on dist-test-slave-zzwf"
server_key: "86f2f19ac948bf51a8a7c5ed711070d3"
server_key_iv: "c0daea5a53a7ec262d3abacbd87ab597"
server_key_version: "encryptionkey@0"
I20260620 20:26:40.512928 16787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:26:40.536533 16787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:26:40.540630 16787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:26:40.540922 16787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:26:40.559900 16787 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:39161
I20260620 20:26:40.559952 16853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:39161 every 8 connection(s)
I20260620 20:26:40.561430 16787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:26:40.565860 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 16787
I20260620 20:26:40.566633 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:45307
--local_ip_for_outbound_sockets=127.31.235.129
--tserver_master_addrs=127.31.235.190:39161
--webserver_port=35663
--webserver_interface=127.31.235.129
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:26:40.568256 16854 sys_catalog.cc:263] Verifying existing consensus state
I20260620 20:26:40.571494 16854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6: Bootstrap starting.
I20260620 20:26:40.601195 16854 log.cc:824] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6: Log is configured to *not* fsync() on all Append() calls
I20260620 20:26:40.622138 16854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:26:40.622850 16854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6: Bootstrap complete.
I20260620 20:26:40.636248 16854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f76da362154684a3f612ac76e371b6" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39161 } }
I20260620 20:26:40.637288 16854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37f76da362154684a3f612ac76e371b6, State: Initialized, Role: FOLLOWER
I20260620 20:26:40.638046 16854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f76da362154684a3f612ac76e371b6" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39161 } }
I20260620 20:26:40.638286 16854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:26:40.638399 16854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:26:40.638579 16854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 2 FOLLOWER]: Advancing to term 3
I20260620 20:26:40.642444 16854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f76da362154684a3f612ac76e371b6" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39161 } }
I20260620 20:26:40.642931 16854 leader_election.cc:302] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37f76da362154684a3f612ac76e371b6; no voters:
I20260620 20:26:40.643610 16854 leader_election.cc:288] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 20:26:40.643756 16858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 20:26:40.644783 16858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [term 3 LEADER]: Becoming Leader. State: Replica: 37f76da362154684a3f612ac76e371b6, State: Running, Role: LEADER
I20260620 20:26:40.645303 16858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f76da362154684a3f612ac76e371b6" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39161 } }
I20260620 20:26:40.645810 16854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:26:40.650489 16859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "37f76da362154684a3f612ac76e371b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f76da362154684a3f612ac76e371b6" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39161 } } }
I20260620 20:26:40.650869 16859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [sys.catalog]: This master's current role is: LEADER
I20260620 20:26:40.651448 16860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37f76da362154684a3f612ac76e371b6. Latest consensus state: current_term: 3 leader_uuid: "37f76da362154684a3f612ac76e371b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f76da362154684a3f612ac76e371b6" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 39161 } } }
I20260620 20:26:40.651777 16860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6 [sys.catalog]: This master's current role is: LEADER
I20260620 20:26:40.659225 16865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:26:40.668303 16865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:26:40.670058 16865 catalog_manager.cc:1259] Loaded cluster ID: a44ca722a80f40aabbcf41b0c8ad76d7
I20260620 20:26:40.670250 16865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:26:40.674510 16865 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:26:40.676088 16865 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6: Loaded TSK: 0
I20260620 20:26:40.676322 16865 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 37f76da362154684a3f612ac76e371b6: Loaded TSK: 1
F20260620 20:26:40.677392 16865 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f53cb4df9ad google::LogMessage::Fail() at ??:0
@ 0x7f53cb4e288a google::LogMessage::SendToLog() at ??:0
@ 0x7f53cb4df4de google::LogMessage::Flush() at ??:0
@ 0x7f53cb4e0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f53cf2437d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f53cf23273e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f53e0f121c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f53e0f11783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f53e0f9d2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f53e0f9d0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f53e0fe75e9 std::function<>::operator()() at ??:0
@ 0x7f53e0f15cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f53e0f13c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f53e0f9b610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f53e0f9b439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f53ce12b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f53ce12e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f53ce12e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f53ce109eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f53d13e36db start_thread at ??:0
@ 0x7f53ca37961f clone at ??:0
W20260620 20:26:40.974349 16856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:26:40.974623 16856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:26:40.974786 16856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:26:40.982194 16856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:26:40.982309 16856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:26:40.982491 16856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:26:40.992354 16856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:45307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=35663
--tserver_master_addrs=127.31.235.190:39161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:26:40.993731 16856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:26:40.995428 16856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:26:41.011821 16902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:26:41.012073 16904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:26:41.012647 16901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:26:41.013757 16856 server_base.cc:1034] running on GCE node
I20260620 20:26:41.014382 16856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:26:41.015350 16856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:26:41.016578 16856 hybrid_clock.cc:648] HybridClock initialized: now 1781987201016508 us; error 57 us; skew 500 ppm
I20260620 20:26:41.016943 16856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:26:41 dist-test-slave-zzwf krb5kdc[15705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987201, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:26:41.020820 16856 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:26:41.022344 16856 webserver.cc:460] Webserver started at http://127.31.235.129:35663/ using document root <none> and password file <none>
I20260620 20:26:41.022987 16856 fs_manager.cc:359] Metadata directory not provided
I20260620 20:26:41.023089 16856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:26:41.028260 16856 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:26:41.031240 16911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:41.032687 16856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:26:41.032887 16856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "225f81a50ceb4492b32d74188461b59f"
format_stamp: "Formatted at 2026-06-20 20:25:55 on dist-test-slave-zzwf"
server_key: "4488c6bbdda90d61cd5a939548c80705"
server_key_iv: "8b1e017e2e26aca057ad3eea9628f09a"
server_key_version: "encryptionkey@0"
I20260620 20:26:41.033962 16856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:26:41.050076 16856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:26:41.054857 16856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:26:41.055212 16856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:26:41.056298 16856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:26:41.058384 16918 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 20:26:41.064294 16856 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 20:26:41.064455 16856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260620 20:26:41.064581 16856 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 20:26:41.067727 16856 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 20:26:41.067816 16856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260620 20:26:41.068216 16918 tablet_bootstrap.cc:492] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f: Bootstrap starting.
I20260620 20:26:41.096966 16918 log.cc:824] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:26:41.107206 16856 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:45307
I20260620 20:26:41.107234 17025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:45307 every 8 connection(s)
I20260620 20:26:41.108431 16856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:26:41.110563 16918 tablet_bootstrap.cc:492] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 20:26:41.111071 16918 tablet_bootstrap.cc:492] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f: Bootstrap complete.
I20260620 20:26:41.112365 16918 ts_tablet_manager.cc:1397] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f: Time spent bootstrapping tablet: real 0.044s user 0.021s sys 0.020s
I20260620 20:26:41.113842 17028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:41.111492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:49137)
0620 20:26:41.112436 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:41.112780 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":431,"threads_started":1}
W20260620 20:26:41.114569 17026 heartbeater.cc:644] Failed to heartbeat to 127.31.235.190:39161 (0 consecutive failures): Network error: Failed to ping master at 127.31.235.190:39161: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
I20260620 20:26:41.115599 17028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:41.115173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:57477)
0620 20:26:41.115412 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:41.115485 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 20:26:41.115949 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 16856
I20260620 20:26:41.116714 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:36427
--local_ip_for_outbound_sockets=127.31.235.130
--tserver_master_addrs=127.31.235.190:39161
--webserver_port=43767
--webserver_interface=127.31.235.130
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:26:41.122522 17028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:41.122085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:37833)
0620 20:26:41.122245 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:41.122371 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 20:26:41.122810 17026 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 20:26:41.122555 16918 raft_consensus.cc:348] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } }
I20260620 20:26:41.124029 16918 raft_consensus.cc:729] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 225f81a50ceb4492b32d74188461b59f, State: Initialized, Role: FOLLOWER
I20260620 20:26:41.124728 16918 consensus_queue.cc:260] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } }
I20260620 20:26:41.126564 16918 ts_tablet_manager.cc:1428] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260620 20:26:41.126739 17028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:41.126258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:32869)
0620 20:26:41.126439 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:41.126531 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 20:26:41.127588 17028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:41.127291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:51175)
0620 20:26:41.127421 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:41.127481 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 20:26:41.447422 17030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:26:41.447739 17030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:26:41.447916 17030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:26:41.455049 17030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:26:41.455202 17030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:26:41.455389 17030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:26:41.464895 17030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:36427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=43767
--tserver_master_addrs=127.31.235.190:39161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:26:41.466404 17030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:26:41.467684 17030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:26:41.481019 17035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:26:41.480984 17038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:26:41.480990 17036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:26:41.482061 17030 server_base.cc:1034] running on GCE node
I20260620 20:26:41.482671 17030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:26:41.483676 17030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:26:41.484941 17030 hybrid_clock.cc:648] HybridClock initialized: now 1781987201484882 us; error 52 us; skew 500 ppm
I20260620 20:26:41.485343 17030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:26:41 dist-test-slave-zzwf krb5kdc[15705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987201, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:26:41.488639 17030 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:26:41.490197 17030 webserver.cc:460] Webserver started at http://127.31.235.130:43767/ using document root <none> and password file <none>
I20260620 20:26:41.490824 17030 fs_manager.cc:359] Metadata directory not provided
I20260620 20:26:41.491106 17030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:26:41.496474 17030 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 20:26:41.499228 17045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:41.500744 17030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:26:41.500932 17030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "1e513437eeac457aa8354f95edcbd9ec"
format_stamp: "Formatted at 2026-06-20 20:25:55 on dist-test-slave-zzwf"
server_key: "fb5e88801ae36d4fe06b819cf2966509"
server_key_iv: "0d924090dcd9c20b8de5b97fce19af5b"
server_key_version: "encryptionkey@0"
I20260620 20:26:41.502019 17030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:26:41.523315 17030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:26:41.527642 17030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:26:41.527927 17030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:26:41.528976 17030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:26:41.530511 17030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:26:41.530644 17030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:41.530798 17030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:26:41.530889 17030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:41.565963 17030 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:36427
I20260620 20:26:41.566016 17157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:36427 every 8 connection(s)
I20260620 20:26:41.567420 17030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:26:41.572207 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:41.570369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.130:48305)
0620 20:26:41.571118 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:41.571379 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":291,"threads_started":1}
W20260620 20:26:41.572852 17158 heartbeater.cc:644] Failed to heartbeat to 127.31.235.190:39161 (0 consecutive failures): Network error: Failed to ping master at 127.31.235.190:39161: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
I20260620 20:26:41.573698 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:41.573339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.130:53147)
0620 20:26:41.573474 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:41.573599 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 20:26:41.574275 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 17030
I20260620 20:26:41.574623 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:41.574287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.130:43239)
0620 20:26:41.574455 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:41.574525 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 20:26:41.574927 17158 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 20:26:41.575018 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:36079
--local_ip_for_outbound_sockets=127.31.235.131
--tserver_master_addrs=127.31.235.190:39161
--webserver_port=46867
--webserver_interface=127.31.235.131
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:33261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:26:41.904796 17161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:26:41.905105 17161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:26:41.905285 17161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:26:41.912817 17161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:26:41.912941 17161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:26:41.913154 17161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:26:41.923389 17161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:33261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:36079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=46867
--tserver_master_addrs=127.31.235.190:39161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:26:41.924763 17161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:26:41.926178 17161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:26:41.939719 17166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:26:41.939769 17169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:26:41.939760 17167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:26:41.940714 17161 server_base.cc:1034] running on GCE node
I20260620 20:26:41.941362 17161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:26:41.942318 17161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:26:41.943535 17161 hybrid_clock.cc:648] HybridClock initialized: now 1781987201943493 us; error 24 us; skew 500 ppm
I20260620 20:26:41.943858 17161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:26:41 dist-test-slave-zzwf krb5kdc[15705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987201, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:26:41.946893 17161 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:26:41.948364 17161 webserver.cc:460] Webserver started at http://127.31.235.131:46867/ using document root <none> and password file <none>
I20260620 20:26:41.949029 17161 fs_manager.cc:359] Metadata directory not provided
I20260620 20:26:41.949175 17161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:26:41.954306 17161 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:26:41.956914 17176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:41.958295 17161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:26:41.958462 17161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "f4de56503aca4b3096531ba2ea1a30d5"
format_stamp: "Formatted at 2026-06-20 20:25:56 on dist-test-slave-zzwf"
server_key: "2a794319c1cdb4011cca3c54091d52cf"
server_key_iv: "7ca62692cd477ae84f3f61e22ec8946d"
server_key_version: "encryptionkey@0"
I20260620 20:26:41.959462 17161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:26:41.983331 17161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:26:41.987475 17161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:26:41.987761 17161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:26:41.988866 17161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:26:41.990298 17161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:26:41.990401 17161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:41.990527 17161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:26:41.990604 17161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:42.025272 17161 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:36079
I20260620 20:26:42.025367 17288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:36079 every 8 connection(s)
I20260620 20:26:42.026746 17161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:26:42.029050 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 17161
I20260620 20:26:42.032061 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:42.030008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.131:55189)
0620 20:26:42.030965 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:42.031248 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":337,"threads_started":1}
W20260620 20:26:42.032703 17289 heartbeater.cc:644] Failed to heartbeat to 127.31.235.190:39161 (0 consecutive failures): Network error: Failed to ping master at 127.31.235.190:39161: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
I20260620 20:26:42.033643 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:42.033254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.131:42833)
0620 20:26:42.033401 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:42.033465 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 20:26:42.034485 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:42.034171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.131:55447)
0620 20:26:42.034296 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:42.034375 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 20:26:42.034716 17289 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 20:26:42.129312 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:42.128488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:58755)
0620 20:26:42.128978 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:42.129064 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":234,"threads_started":1}
I20260620 20:26:42.441766 17294 raft_consensus.cc:482] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:26:42.442052 17294 raft_consensus.cc:504] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } }
I20260620 20:26:42.443635 17294 leader_election.cc:288] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e513437eeac457aa8354f95edcbd9ec (127.31.235.130:36427), f4de56503aca4b3096531ba2ea1a30d5 (127.31.235.131:36079)
Jun 20 20:26:42 dist-test-slave-zzwf krb5kdc[15705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987201, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.130@KRBTEST.COM
Jun 20 20:26:42 dist-test-slave-zzwf krb5kdc[15705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987201, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.131@KRBTEST.COM
I20260620 20:26:42.466281 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:42.444779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:40815 (local address 127.31.235.131:36079)
0620 20:26:42.446911 (+ 2132us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:42.446926 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:42.447470 (+ 544us) server_negotiation.cc:407] Connection header received
0620 20:26:42.448983 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:42.448991 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:42.449322 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:42.449641 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:26:42.451397 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:42.452595 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:42.454367 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:42.454981 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:42.455178 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:42.459689 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:26:42.459696 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:26:42.459707 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:26:42.459744 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:26:42.462664 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:42.463392 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:42.463399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:42.463413 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:42.463584 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:42.464049 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:42.464054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:42.464057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:42.464495 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:26:42.464682 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 20:26:42.465458 (+ 776us) server_negotiation.cc:299] Negotiation successful
0620 20:26:42.465827 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1940,"thread_start_us":1764,"threads_started":1}
I20260620 20:26:42.466295 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:42.444142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:39753 (local address 127.31.235.130:36427)
0620 20:26:42.447131 (+ 2989us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:42.447144 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:42.447183 (+ 39us) server_negotiation.cc:407] Connection header received
0620 20:26:42.448746 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:42.448755 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:42.449116 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:42.449433 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:26:42.451422 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:42.452580 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:42.454373 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:42.455021 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:42.455234 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:42.459420 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:26:42.459428 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:26:42.459440 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:26:42.459478 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:26:42.462503 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:42.463355 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:42.463362 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:42.463367 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:42.463523 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:42.464001 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:42.464005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:42.464007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:42.464489 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:26:42.464683 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 20:26:42.465458 (+ 775us) server_negotiation.cc:299] Negotiation successful
0620 20:26:42.465860 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2810,"thread_start_us":2703,"threads_started":1}
I20260620 20:26:42.466287 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:42.443958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:36427 (local address 127.31.235.129:39753)
0620 20:26:42.444125 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:42.444157 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 20:26:42.444760 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:26:42.449571 (+ 4811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:26:42.449580 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:26:42.450486 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:26:42.451190 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:42.451248 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:42.452781 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:42.452784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:42.454227 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:42.454239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:42.455241 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:42.455248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:42.455480 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:42.456350 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:26:42.456401 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:26:42.459128 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:26:42.462644 (+ 3516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:26:42.462654 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:26:42.462668 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:26:42.463207 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:26:42.463649 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:26:42.463654 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:26:42.463658 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:26:42.463860 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:26:42.464626 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:26:42.464633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:26:42.464964 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 20:26:42.465237 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 20:26:42.465824 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":322,"spinlock_wait_cycles":24576}
I20260620 20:26:42.466425 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:42.444417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:36079 (local address 127.31.235.129:40815)
0620 20:26:42.447349 (+ 2932us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:42.447366 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:26:42.447496 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:26:42.449594 (+ 2098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:26:42.449599 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:26:42.450482 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:26:42.451190 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:42.451210 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:42.452771 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:42.452775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:42.454231 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:42.454243 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:42.455171 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:42.455178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:42.455477 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:42.456349 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:26:42.456397 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:26:42.459450 (+ 3053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:26:42.462850 (+ 3400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:26:42.462854 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:26:42.462857 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:26:42.463259 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:26:42.463704 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:26:42.463713 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:26:42.463715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:26:42.463907 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:26:42.464630 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:26:42.464634 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:26:42.464963 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 20:26:42.465237 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 20:26:42.465977 (+ 740us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2810,"mutex_wait_us":443,"spinlock_wait_cycles":48128,"thread_start_us":2608,"threads_started":1}
W20260620 20:26:42.469610 16912 leader_election.cc:341] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e513437eeac457aa8354f95edcbd9ec (127.31.235.130:36427): Not found: Tablet not found: da81de0ad9df40d9bb6f594306a31f74
W20260620 20:26:42.469931 16912 leader_election.cc:341] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4de56503aca4b3096531ba2ea1a30d5 (127.31.235.131:36079): Not found: Tablet not found: da81de0ad9df40d9bb6f594306a31f74
I20260620 20:26:42.470080 16912 leader_election.cc:302] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 225f81a50ceb4492b32d74188461b59f; no voters: 1e513437eeac457aa8354f95edcbd9ec, f4de56503aca4b3096531ba2ea1a30d5
I20260620 20:26:42.470490 17294 raft_consensus.cc:2726] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:26:42.576268 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:42.575466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.130:52589)
0620 20:26:42.575896 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:42.576041 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":202,"threads_started":1}
I20260620 20:26:43.036067 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:43.035287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.131:37247)
0620 20:26:43.035743 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:43.035874 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":225,"threads_started":1}
I20260620 20:26:43.131294 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:43.130468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:54309)
0620 20:26:43.131034 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:43.131123 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":293,"threads_started":1}
I20260620 20:26:43.577934 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:43.577190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.130:50085)
0620 20:26:43.577687 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:43.577773 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":240,"threads_started":1}
I20260620 20:26:44.037784 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:44.036944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.131:42099)
0620 20:26:44.037466 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:44.037610 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":292,"threads_started":1}
I20260620 20:26:44.133181 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:44.132331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:52137)
0620 20:26:44.132908 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:44.132996 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":258,"threads_started":1}
I20260620 20:26:44.579313 17304 raft_consensus.cc:482] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:26:44.579543 17304 raft_consensus.cc:504] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } }
I20260620 20:26:44.579716 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:44.578850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.130:51009)
0620 20:26:44.579413 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:44.579526 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":320,"threads_started":1}
I20260620 20:26:44.580284 17304 leader_election.cc:288] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e513437eeac457aa8354f95edcbd9ec (127.31.235.130:36427), f4de56503aca4b3096531ba2ea1a30d5 (127.31.235.131:36079)
W20260620 20:26:44.581300 16912 leader_election.cc:341] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e513437eeac457aa8354f95edcbd9ec (127.31.235.130:36427): Not found: Tablet not found: da81de0ad9df40d9bb6f594306a31f74
W20260620 20:26:44.581599 16912 leader_election.cc:341] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4de56503aca4b3096531ba2ea1a30d5 (127.31.235.131:36079): Not found: Tablet not found: da81de0ad9df40d9bb6f594306a31f74
I20260620 20:26:44.581710 16912 leader_election.cc:302] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 225f81a50ceb4492b32d74188461b59f; no voters: 1e513437eeac457aa8354f95edcbd9ec, f4de56503aca4b3096531ba2ea1a30d5
I20260620 20:26:44.582016 17304 raft_consensus.cc:2726] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:26:45.039608 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:45.038825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.131:33989)
0620 20:26:45.039341 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:45.039429 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":250,"threads_started":1}
I20260620 20:26:45.135185 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:45.134410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:39293)
0620 20:26:45.134898 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:45.134990 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":222,"threads_started":1}
I20260620 20:26:45.581444 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:45.580647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.130:42123)
0620 20:26:45.581117 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:45.581251 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":241,"threads_started":1}
I20260620 20:26:46.041419 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:46.040638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.131:48987)
0620 20:26:46.041080 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:46.041176 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":203,"threads_started":1}
I20260620 20:26:46.137182 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:46.136417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:59875)
0620 20:26:46.136927 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:46.137010 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":255,"threads_started":1}
I20260620 20:26:46.202699 17311 raft_consensus.cc:482] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:26:46.202981 17311 raft_consensus.cc:504] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } }
I20260620 20:26:46.203701 17311 leader_election.cc:288] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e513437eeac457aa8354f95edcbd9ec (127.31.235.130:36427), f4de56503aca4b3096531ba2ea1a30d5 (127.31.235.131:36079)
W20260620 20:26:46.204738 16912 leader_election.cc:341] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e513437eeac457aa8354f95edcbd9ec (127.31.235.130:36427): Not found: Tablet not found: da81de0ad9df40d9bb6f594306a31f74
W20260620 20:26:46.204980 16912 leader_election.cc:341] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4de56503aca4b3096531ba2ea1a30d5 (127.31.235.131:36079): Not found: Tablet not found: da81de0ad9df40d9bb6f594306a31f74
I20260620 20:26:46.205128 16912 leader_election.cc:302] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 225f81a50ceb4492b32d74188461b59f; no voters: 1e513437eeac457aa8354f95edcbd9ec, f4de56503aca4b3096531ba2ea1a30d5
I20260620 20:26:46.205408 17311 raft_consensus.cc:2726] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:26:46.583119 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:46.582380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.130:38377)
0620 20:26:46.582864 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:46.582949 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":255,"threads_started":1}
I20260620 20:26:47.043115 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:47.042421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.131:32893)
0620 20:26:47.042885 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:47.042966 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":268,"threads_started":1}
I20260620 20:26:47.139000 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:47.138283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:39955)
0620 20:26:47.138722 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:47.138838 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":238,"threads_started":1}
I20260620 20:26:47.584883 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:47.584119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.130:39467)
0620 20:26:47.584612 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:47.584698 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":256,"threads_started":1}
I20260620 20:26:48.044667 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:48.043953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.131:49717)
0620 20:26:48.044413 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:48.044491 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":254,"threads_started":1}
I20260620 20:26:48.140709 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:48.139900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:40937)
0620 20:26:48.140399 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:48.140518 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":280,"threads_started":1}
I20260620 20:26:48.586874 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:48.586076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.130:42345)
0620 20:26:48.586611 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:48.586694 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":255,"threads_started":1}
I20260620 20:26:49.046618 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:49.045735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.131:39141)
0620 20:26:49.046291 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:49.046385 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":239,"threads_started":1}
I20260620 20:26:49.142454 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:49.141729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:55639)
0620 20:26:49.142206 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:49.142298 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":257,"threads_started":1}
I20260620 20:26:49.588502 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:49.587811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.130:57481)
0620 20:26:49.588236 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:49.588317 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":217,"threads_started":1}
I20260620 20:26:50.048372 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:50.047632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.131:34175)
0620 20:26:50.048079 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:50.048168 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":228,"threads_started":1}
I20260620 20:26:50.144073 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:50.143365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:33899)
0620 20:26:50.143804 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:50.143892 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":221,"threads_started":1}
I20260620 20:26:50.590284 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:50.589461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.130:59515)
0620 20:26:50.590010 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:50.590120 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":252,"threads_started":1}
I20260620 20:26:51.050149 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:51.049350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.131:56951)
0620 20:26:51.049902 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:51.049996 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":313,"threads_started":1}
I20260620 20:26:51.071843 17326 raft_consensus.cc:482] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:26:51.072023 17326 raft_consensus.cc:504] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } }
I20260620 20:26:51.072571 17326 leader_election.cc:288] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e513437eeac457aa8354f95edcbd9ec (127.31.235.130:36427), f4de56503aca4b3096531ba2ea1a30d5 (127.31.235.131:36079)
W20260620 20:26:51.073643 16912 leader_election.cc:341] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4de56503aca4b3096531ba2ea1a30d5 (127.31.235.131:36079): Not found: Tablet not found: da81de0ad9df40d9bb6f594306a31f74
W20260620 20:26:51.073870 16912 leader_election.cc:341] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e513437eeac457aa8354f95edcbd9ec (127.31.235.130:36427): Not found: Tablet not found: da81de0ad9df40d9bb6f594306a31f74
I20260620 20:26:51.074003 16912 leader_election.cc:302] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 225f81a50ceb4492b32d74188461b59f; no voters: 1e513437eeac457aa8354f95edcbd9ec, f4de56503aca4b3096531ba2ea1a30d5
I20260620 20:26:51.074319 17326 raft_consensus.cc:2726] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:26:51.145995 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:51.145109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:59711)
0620 20:26:51.145678 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:51.145800 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":268,"threads_started":1}
I20260620 20:26:51.592031 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:51.591311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.130:60037)
0620 20:26:51.591763 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:51.591846 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":242,"threads_started":1}
I20260620 20:26:52.051774 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:52.051018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.131:44529)
0620 20:26:52.051497 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:52.051587 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":257,"threads_started":1}
I20260620 20:26:52.147603 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:52.146879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:51081)
0620 20:26:52.147334 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:52.147416 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":228,"threads_started":1}
I20260620 20:26:52.593881 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:52.593016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.130:49525)
0620 20:26:52.593622 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:52.593715 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":288,"threads_started":1}
I20260620 20:26:53.053622 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:53.052850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.131:37113)
0620 20:26:53.053306 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:53.053390 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":214,"threads_started":1}
I20260620 20:26:53.149268 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:53.148562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:42093)
0620 20:26:53.149021 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:53.149108 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":230,"threads_started":1}
I20260620 20:26:53.595561 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:53.594828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.130:58315)
0620 20:26:53.595298 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:53.595381 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":251,"threads_started":1}
I20260620 20:26:54.055431 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:54.054576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.131:49813)
0620 20:26:54.055124 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:54.055234 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":302,"threads_started":1}
I20260620 20:26:54.150909 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:54.150171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:60551)
0620 20:26:54.150615 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:54.150702 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":237,"threads_started":1}
I20260620 20:26:54.597275 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:54.596499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.130:36553)
0620 20:26:54.597010 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:54.597110 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":263,"threads_started":1}
I20260620 20:26:54.860726 17338 raft_consensus.cc:482] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:26:54.860915 17338 raft_consensus.cc:504] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "225f81a50ceb4492b32d74188461b59f" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45307 } } peers { permanent_uuid: "1e513437eeac457aa8354f95edcbd9ec" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36427 } } peers { permanent_uuid: "f4de56503aca4b3096531ba2ea1a30d5" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36079 } }
I20260620 20:26:54.861601 17338 leader_election.cc:288] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1e513437eeac457aa8354f95edcbd9ec (127.31.235.130:36427), f4de56503aca4b3096531ba2ea1a30d5 (127.31.235.131:36079)
W20260620 20:26:54.862612 16912 leader_election.cc:341] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e513437eeac457aa8354f95edcbd9ec (127.31.235.130:36427): Not found: Tablet not found: da81de0ad9df40d9bb6f594306a31f74
W20260620 20:26:54.862839 16912 leader_election.cc:341] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4de56503aca4b3096531ba2ea1a30d5 (127.31.235.131:36079): Not found: Tablet not found: da81de0ad9df40d9bb6f594306a31f74
I20260620 20:26:54.862993 16912 leader_election.cc:302] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 225f81a50ceb4492b32d74188461b59f; no voters: 1e513437eeac457aa8354f95edcbd9ec, f4de56503aca4b3096531ba2ea1a30d5
I20260620 20:26:54.863256 17338 raft_consensus.cc:2726] T da81de0ad9df40d9bb6f594306a31f74 P 225f81a50ceb4492b32d74188461b59f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 20:26:55.056962 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:55.056248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.131:53729)
0620 20:26:55.056708 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:55.056793 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":254,"threads_started":1}
I20260620 20:26:55.152827 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:55.151933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:34079)
0620 20:26:55.152488 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:55.152581 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":273,"threads_started":1}
I20260620 20:26:55.598932 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:55.598170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.130:39805)
0620 20:26:55.598641 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:55.598732 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":233,"threads_started":1}
I20260620 20:26:56.058624 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:56.057878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.131:33907)
0620 20:26:56.058339 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:56.058423 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":221,"threads_started":1}
I20260620 20:26:56.154747 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:56.153996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.129:36477)
0620 20:26:56.154463 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:56.154575 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":228,"threads_started":1}
I20260620 20:26:56.600756 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:56.600043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.130:55479)
0620 20:26:56.600491 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:56.600595 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":228,"threads_started":1}
I20260620 20:26:57.037842 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 16856
I20260620 20:26:57.053146 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:57.038945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50644 (local address 127.31.235.129:45307)
0620 20:26:57.039471 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:57.039482 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:57.039519 (+ 37us) server_negotiation.cc:407] Connection header received
0620 20:26:57.040030 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:57.040038 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:57.040263 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:57.040423 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:26:57.041842 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:57.042823 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:57.044049 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:57.044596 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:57.044693 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:57.047806 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:26:57.047819 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:26:57.047837 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:26:57.047876 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:26:57.050622 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:57.051250 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:57.051256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:57.051259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:57.051405 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:57.051721 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:57.051724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:57.051727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:57.052103 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:26:57.052358 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 20:26:57.052569 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 20:26:57.052764 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":283,"threads_started":1}
I20260620 20:26:57.054471 17021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:26:57.060410 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:57.059642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:39161 (local address 127.31.235.131:36449)
0620 20:26:57.060154 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:57.060254 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.190:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":269,"threads_started":1}
I20260620 20:26:57.145040 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 16856
I20260620 20:26:57.162508 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 17030
I20260620 20:26:57.175732 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:57.163263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36832 (local address 127.31.235.130:36427)
0620 20:26:57.163757 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:57.163761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:57.163780 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:26:57.163838 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:57.163841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:57.163902 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:57.163993 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:26:57.165405 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:57.166230 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:57.167508 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:57.167936 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:57.168050 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:57.170560 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:26:57.170566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:26:57.170570 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:26:57.170604 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:26:57.173310 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:57.173939 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:57.173943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:57.173946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:57.174078 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:57.174386 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:57.174389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:57.174391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:57.174660 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:26:57.174871 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 20:26:57.175137 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 20:26:57.175335 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":275,"threads_started":1}
I20260620 20:26:57.176535 17153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:26:57.260282 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 17030
I20260620 20:26:57.277928 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 17161
I20260620 20:26:57.291718 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:57.278776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38336 (local address 127.31.235.131:36079)
0620 20:26:57.279339 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:57.279344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:57.279360 (+ 16us) server_negotiation.cc:407] Connection header received
0620 20:26:57.279416 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:57.279420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:57.279489 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:57.279609 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:26:57.280989 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:57.281851 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:57.283107 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:57.283542 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:57.283677 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:57.286454 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:26:57.286460 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:26:57.286463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:26:57.286496 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:26:57.289010 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:57.289799 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:57.289804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:57.289808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:57.289952 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:57.290333 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:57.290342 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:57.290344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:57.290592 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:26:57.290880 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 20:26:57.291118 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 20:26:57.291334 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":250,"threads_started":1}
I20260620 20:26:57.292435 17284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:26:57.377063 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 17161
2026-06-20T20:26:57Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64625 ms)
[----------] 30 tests from SecurityITest (300732 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:26:57 dist-test-slave-zzwf krb5kdc[17354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:26:57 dist-test-slave-zzwf krb5kdc[17354](info): set up 2 sockets
Jun 20 20:26:57 dist-test-slave-zzwf krb5kdc[17354](info): commencing operation
krb5kdc: starting...
W20260620 20:26:58.490068 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:26:58 dist-test-slave-zzwf krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:26:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:26:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:26:58.741271 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:41693
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:43093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:41693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:26:59.076195 17367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:26:59.076601 17367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:26:59.076753 17367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:26:59.084137 17367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:26:59.084297 17367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:26:59.084394 17367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:26:59.084493 17367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:26:59.084566 17367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:26:59.094696 17367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:41693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:41693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:26:59.096052 17367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:26:59.097371 17367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:26:59.109790 17372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:26:59.109722 17373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:26:59.109696 17375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:26:59.110394 17367 server_base.cc:1034] running on GCE node
I20260620 20:26:59.111372 17367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:26:59.112792 17367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:26:59.114017 17367 hybrid_clock.cc:648] HybridClock initialized: now 1781987219113955 us; error 60 us; skew 500 ppm
I20260620 20:26:59.114410 17367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:26:59 dist-test-slave-zzwf krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987219, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:26:59.117654 17367 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:26:59.119088 17367 webserver.cc:460] Webserver started at http://127.31.235.190:35885/ using document root <none> and password file <none>
I20260620 20:26:59.119841 17367 fs_manager.cc:359] Metadata directory not provided
I20260620 20:26:59.119987 17367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:26:59.120281 17367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:26:59.123292 17367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "15aa01658b844f1c8ab690b85bca3fdb"
format_stamp: "Formatted at 2026-06-20 20:26:59 on dist-test-slave-zzwf"
server_key: "516cbb4e8f427c8067dfdbb37fdb551b"
server_key_iv: "d011ca1265be051b3306931b0cf974ab"
server_key_version: "encryptionkey@0"
I20260620 20:26:59.124069 17367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "15aa01658b844f1c8ab690b85bca3fdb"
format_stamp: "Formatted at 2026-06-20 20:26:59 on dist-test-slave-zzwf"
server_key: "516cbb4e8f427c8067dfdbb37fdb551b"
server_key_iv: "d011ca1265be051b3306931b0cf974ab"
server_key_version: "encryptionkey@0"
I20260620 20:26:59.129287 17367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 20:26:59.132653 17382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:59.133998 17367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:26:59.134177 17367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "15aa01658b844f1c8ab690b85bca3fdb"
format_stamp: "Formatted at 2026-06-20 20:26:59 on dist-test-slave-zzwf"
server_key: "516cbb4e8f427c8067dfdbb37fdb551b"
server_key_iv: "d011ca1265be051b3306931b0cf974ab"
server_key_version: "encryptionkey@0"
I20260620 20:26:59.134371 17367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:26:59.151361 17367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:26:59.155753 17367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:26:59.156036 17367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:26:59.174803 17367 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:41693
I20260620 20:26:59.174829 17433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:41693 every 8 connection(s)
I20260620 20:26:59.176143 17367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:26:59.179823 17434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:26:59.184767 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 17367
I20260620 20:26:59.184989 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:26:59.185668 32686 external_mini_cluster.cc:1428] Setting key 7b469164a56856aa4df5f19955f17f31
I20260620 20:26:59.190564 17434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb: Bootstrap starting.
I20260620 20:26:59.194284 17434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb: Neither blocks nor log segments found. Creating new log.
I20260620 20:26:59.195642 17434 log.cc:824] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb: Log is configured to *not* fsync() on all Append() calls
I20260620 20:26:59.199326 17434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb: No bootstrap required, opened a new log
Jun 20 20:26:59 dist-test-slave-zzwf krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:26:59.208117 17434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aa01658b844f1c8ab690b85bca3fdb" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 41693 } }
I20260620 20:26:59.208513 17434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:26:59.208644 17434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15aa01658b844f1c8ab690b85bca3fdb, State: Initialized, Role: FOLLOWER
I20260620 20:26:59.209237 17434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aa01658b844f1c8ab690b85bca3fdb" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 41693 } }
I20260620 20:26:59.209556 17434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:26:59.209728 17434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:26:59.209836 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:59.187600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54082 (local address 127.31.235.190:41693)
0620 20:26:59.188513 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:59.188524 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:59.188567 (+ 43us) server_negotiation.cc:407] Connection header received
0620 20:26:59.190283 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:59.190303 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:59.190701 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:59.191023 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:26:59.192883 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:59.194206 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:59.196161 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:59.197224 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:59.197352 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:59.202519 (+ 5167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:26:59.202529 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:26:59.202543 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:26:59.202586 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:26:59.205697 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:59.206390 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:59.206396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:59.206402 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:59.206564 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:59.206943 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:59.206949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:59.206952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:59.207479 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:26:59.207774 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 20:26:59.208196 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 20:26:59.208647 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":428,"threads_started":1}
I20260620 20:26:59.209961 17434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:26:59.212939 17434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aa01658b844f1c8ab690b85bca3fdb" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 41693 } }
I20260620 20:26:59.213554 17434 leader_election.cc:302] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15aa01658b844f1c8ab690b85bca3fdb; no voters:
I20260620 20:26:59.214155 17434 leader_election.cc:288] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:26:59.214255 17439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:26:59.215021 17439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb [term 1 LEADER]: Becoming Leader. State: Replica: 15aa01658b844f1c8ab690b85bca3fdb, State: Running, Role: LEADER
I20260620 20:26:59.215541 17439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aa01658b844f1c8ab690b85bca3fdb" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 41693 } }
I20260620 20:26:59.216353 17434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:26:59.218442 17441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15aa01658b844f1c8ab690b85bca3fdb. Latest consensus state: current_term: 1 leader_uuid: "15aa01658b844f1c8ab690b85bca3fdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aa01658b844f1c8ab690b85bca3fdb" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 41693 } } }
I20260620 20:26:59.218958 17441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb [sys.catalog]: This master's current role is: LEADER
I20260620 20:26:59.219671 17440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15aa01658b844f1c8ab690b85bca3fdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aa01658b844f1c8ab690b85bca3fdb" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 41693 } } }
I20260620 20:26:59.220093 17440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb [sys.catalog]: This master's current role is: LEADER
I20260620 20:26:59.221764 17446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:26:59.227838 17446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:26:59.236320 17446 catalog_manager.cc:1347] Generated new cluster ID: 5808afc0f621470e9e777f9658196762
I20260620 20:26:59.236461 17446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:26:59.249362 17446 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:26:59.251178 17446 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:26:59.258461 17446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb: Generated new TSK 0
I20260620 20:26:59.259512 17446 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
I20260620 20:26:59.323498 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:41693
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:43093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:26:59.657891 17460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:26:59.658190 17460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:26:59.658330 17460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:26:59.665601 17460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:26:59.665712 17460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:26:59.665832 17460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:26:59.675539 17460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:41693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:26:59.676788 17460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:26:59.678046 17460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:26:59.691085 17465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:26:59.691140 17468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:26:59.691177 17466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:26:59.691704 17460 server_base.cc:1034] running on GCE node
I20260620 20:26:59.692373 17460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:26:59.693297 17460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:26:59.694522 17460 hybrid_clock.cc:648] HybridClock initialized: now 1781987219694458 us; error 51 us; skew 500 ppm
I20260620 20:26:59.694921 17460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:26:59 dist-test-slave-zzwf krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987219, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:26:59.698199 17460 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:26:59.699733 17460 webserver.cc:460] Webserver started at http://127.31.235.129:42939/ using document root <none> and password file <none>
I20260620 20:26:59.700387 17460 fs_manager.cc:359] Metadata directory not provided
I20260620 20:26:59.700490 17460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:26:59.700742 17460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:26:59.703667 17460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "9c99da3b4ed7410f8e3245a5162387ea"
format_stamp: "Formatted at 2026-06-20 20:26:59 on dist-test-slave-zzwf"
server_key: "c861a7629605e7b38327ba3b6665586b"
server_key_iv: "4bb2e64adddcb078986be48205f0ab68"
server_key_version: "encryptionkey@0"
I20260620 20:26:59.704489 17460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "9c99da3b4ed7410f8e3245a5162387ea"
format_stamp: "Formatted at 2026-06-20 20:26:59 on dist-test-slave-zzwf"
server_key: "c861a7629605e7b38327ba3b6665586b"
server_key_iv: "4bb2e64adddcb078986be48205f0ab68"
server_key_version: "encryptionkey@0"
I20260620 20:26:59.709784 17460 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 20:26:59.713943 17475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:59.715699 17460 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 20:26:59.715901 17460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "9c99da3b4ed7410f8e3245a5162387ea"
format_stamp: "Formatted at 2026-06-20 20:26:59 on dist-test-slave-zzwf"
server_key: "c861a7629605e7b38327ba3b6665586b"
server_key_iv: "4bb2e64adddcb078986be48205f0ab68"
server_key_version: "encryptionkey@0"
I20260620 20:26:59.716140 17460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:26:59.734108 17460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:26:59.738095 17460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:26:59.738373 17460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:26:59.739321 17460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:26:59.740649 17460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:26:59.740759 17460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:59.740900 17460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:26:59.740970 17460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:26:59.772303 17460 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:41123
I20260620 20:26:59.772336 17587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:41123 every 8 connection(s)
I20260620 20:26:59.773633 17460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:26:59.780479 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 17460
I20260620 20:26:59.780687 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:26:59.781373 32686 external_mini_cluster.cc:1428] Setting key e24b8d48bc2fcd99a90d90114c4f7241
Jun 20 20:26:59 dist-test-slave-zzwf krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987219, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:26:59.795571 17590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:59.776120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:60567 (local address 127.31.235.190:41693)
0620 20:26:59.776574 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 20:26:59.776578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:26:59.777587 (+ 1009us) server_negotiation.cc:407] Connection header received
0620 20:26:59.779439 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:26:59.779444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:26:59.779518 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:26:59.779615 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:26:59.781228 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:59.782270 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:59.783903 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:26:59.784597 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:26:59.784744 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:59.788474 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:26:59.788484 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:26:59.788492 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:26:59.788538 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:26:59.792107 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:59.793117 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:59.793122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:59.793124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:59.793252 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:26:59.793809 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:26:59.793812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:26:59.793814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:26:59.794098 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:26:59.794212 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 20:26:59.795120 (+ 908us) server_negotiation.cc:299] Negotiation successful
0620 20:26:59.795304 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":219,"threads_started":1}
I20260620 20:26:59.797158 17591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:26:59.776667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:41693 (local address 127.31.235.129:60567)
0620 20:26:59.777389 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0620 20:26:59.777436 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 20:26:59.779184 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:26:59.779821 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:26:59.779830 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:26:59.780372 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:26:59.781043 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:59.781062 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:59.782442 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:59.782447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:59.783750 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:26:59.783763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:26:59.784825 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:26:59.784830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:26:59.785031 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:26:59.785778 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:26:59.785802 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:26:59.788246 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:26:59.792258 (+ 4012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:26:59.792269 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:26:59.792291 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:26:59.792977 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:26:59.793373 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:26:59.793381 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:26:59.793387 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:26:59.793663 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:26:59.794222 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:26:59.794233 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:26:59.794719 (+ 486us) client_negotiation.cc:769] Sending connection context
0620 20:26:59.795032 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 20:26:59.795627 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":289,"threads_started":1}
I20260620 20:26:59.798882 17588 heartbeater.cc:347] Connected to a master server at 127.31.235.190:41693
I20260620 20:26:59.799180 17588 heartbeater.cc:464] Registering TS with master...
I20260620 20:26:59.799947 17588 heartbeater.cc:511] Master 127.31.235.190:41693 requested a full tablet report, sending...
I20260620 20:26:59.802193 17399 ts_manager.cc:194] Registered new tserver with Master: 9c99da3b4ed7410f8e3245a5162387ea (127.31.235.129:41123)
I20260620 20:26:59.804466 17399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:60567
WARNING: no policy specified for kudu/127.31.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
I20260620 20:26:59.841701 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:41693
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:43093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:00.175199 17594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:00.175510 17594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:00.175664 17594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:00.183164 17594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:00.183316 17594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:00.183506 17594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:27:00.193868 17594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:41693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:00.195261 17594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:00.196975 17594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:00.210390 17600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:00.210484 17599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:00.210458 17602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:00.211508 17594 server_base.cc:1034] running on GCE node
I20260620 20:27:00.212116 17594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:00.213161 17594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:00.214406 17594 hybrid_clock.cc:648] HybridClock initialized: now 1781987220214347 us; error 65 us; skew 500 ppm
I20260620 20:27:00.214736 17594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:00 dist-test-slave-zzwf krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987220, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:00.218024 17594 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:27:00.219460 17594 webserver.cc:460] Webserver started at http://127.31.235.130:36131/ using document root <none> and password file <none>
I20260620 20:27:00.220117 17594 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:00.220223 17594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:00.220480 17594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:00.223522 17594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "ba5b4915cac741838e7c8d211172794c"
format_stamp: "Formatted at 2026-06-20 20:27:00 on dist-test-slave-zzwf"
server_key: "53b66d6cc3f9f2db1e7c5a0720c89ec8"
server_key_iv: "5a3a399bfe83664a319246842b92eda7"
server_key_version: "encryptionkey@0"
I20260620 20:27:00.224305 17594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "ba5b4915cac741838e7c8d211172794c"
format_stamp: "Formatted at 2026-06-20 20:27:00 on dist-test-slave-zzwf"
server_key: "53b66d6cc3f9f2db1e7c5a0720c89ec8"
server_key_iv: "5a3a399bfe83664a319246842b92eda7"
server_key_version: "encryptionkey@0"
I20260620 20:27:00.229424 17594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 20:27:00.232947 17609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:00.234426 17594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:27:00.234567 17594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "ba5b4915cac741838e7c8d211172794c"
format_stamp: "Formatted at 2026-06-20 20:27:00 on dist-test-slave-zzwf"
server_key: "53b66d6cc3f9f2db1e7c5a0720c89ec8"
server_key_iv: "5a3a399bfe83664a319246842b92eda7"
server_key_version: "encryptionkey@0"
I20260620 20:27:00.234746 17594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:00.274480 17594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:00.278661 17594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:00.278903 17594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:00.279845 17594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:27:00.281132 17594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:27:00.281212 17594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:00.281292 17594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:27:00.281335 17594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:00.314814 17594 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:44209
I20260620 20:27:00.314898 17721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:44209 every 8 connection(s)
I20260620 20:27:00.316165 17594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:27:00.325845 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 17594
I20260620 20:27:00.326030 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:27:00.326604 32686 external_mini_cluster.cc:1428] Setting key 799c4746e9d3d8f13456702d0ae2b4e2
Jun 20 20:27:00 dist-test-slave-zzwf krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987220, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:27:00.336360 17724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:00.318761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:47795 (local address 127.31.235.190:41693)
0620 20:27:00.319245 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:00.319250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:00.320136 (+ 886us) server_negotiation.cc:407] Connection header received
0620 20:27:00.322160 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:00.322170 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:00.322270 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:00.322391 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:00.323894 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.324599 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.326562 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.327057 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.327209 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:00.330765 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:00.330770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:00.330773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:00.330806 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:00.333323 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:00.334086 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:00.334090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:00.334093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:00.334213 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:00.334632 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:00.334637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:00.334640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:00.334915 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:00.335075 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:00.335736 (+ 661us) server_negotiation.cc:299] Negotiation successful
0620 20:27:00.335954 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":243,"threads_started":1}
I20260620 20:27:00.337577 17725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:00.319177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:41693 (local address 127.31.235.130:47795)
0620 20:27:00.319982 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:00.320019 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:00.321800 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:00.322621 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:00.322630 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:00.323163 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:00.323703 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:00.323736 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.324753 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.324757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:00.326418 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:00.326435 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.327286 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.327291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:00.327592 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:00.328339 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:27:00.328370 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:00.330535 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:00.333481 (+ 2946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:00.333518 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:00.333535 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:00.333956 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:00.334322 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:00.334328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:00.334332 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:00.334517 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:00.335038 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:00.335046 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:00.335386 (+ 340us) client_negotiation.cc:769] Sending connection context
0620 20:27:00.335642 (+ 256us) client_negotiation.cc:240] Negotiation successful
0620 20:27:00.336165 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":313,"threads_started":1}
I20260620 20:27:00.339229 17722 heartbeater.cc:347] Connected to a master server at 127.31.235.190:41693
I20260620 20:27:00.339555 17722 heartbeater.cc:464] Registering TS with master...
I20260620 20:27:00.340251 17722 heartbeater.cc:511] Master 127.31.235.190:41693 requested a full tablet report, sending...
I20260620 20:27:00.341725 17399 ts_manager.cc:194] Registered new tserver with Master: ba5b4915cac741838e7c8d211172794c (127.31.235.130:44209)
I20260620 20:27:00.342528 17399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:47795
WARNING: no policy specified for kudu/127.31.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
I20260620 20:27:00.380981 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:41693
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:43093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:00.705952 17728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:00.706235 17728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:00.706342 17728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:00.713639 17728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:00.713754 17728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:00.713867 17728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:27:00.725679 17728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:41693
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:00.726907 17728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:00.728169 17728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:00.741286 17733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:00.741339 17734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:00.741365 17736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:00.742193 17728 server_base.cc:1034] running on GCE node
I20260620 20:27:00.742831 17728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:00.743788 17728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:00.744987 17728 hybrid_clock.cc:648] HybridClock initialized: now 1781987220744938 us; error 42 us; skew 500 ppm
I20260620 20:27:00.745301 17728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:00 dist-test-slave-zzwf krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987220, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:00.748353 17728 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:27:00.749872 17728 webserver.cc:460] Webserver started at http://127.31.235.131:41231/ using document root <none> and password file <none>
I20260620 20:27:00.750546 17728 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:00.750679 17728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:00.750980 17728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:00.753973 17728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "ba58824857444423a9a15fa0651ad228"
format_stamp: "Formatted at 2026-06-20 20:27:00 on dist-test-slave-zzwf"
server_key: "ed50584863903f5e5c87959d4b8abd5b"
server_key_iv: "113240a36d9594a2254389e91d523d25"
server_key_version: "encryptionkey@0"
I20260620 20:27:00.754714 17728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "ba58824857444423a9a15fa0651ad228"
format_stamp: "Formatted at 2026-06-20 20:27:00 on dist-test-slave-zzwf"
server_key: "ed50584863903f5e5c87959d4b8abd5b"
server_key_iv: "113240a36d9594a2254389e91d523d25"
server_key_version: "encryptionkey@0"
I20260620 20:27:00.759811 17728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:27:00.763271 17743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:00.764559 17728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:27:00.764729 17728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "ba58824857444423a9a15fa0651ad228"
format_stamp: "Formatted at 2026-06-20 20:27:00 on dist-test-slave-zzwf"
server_key: "ed50584863903f5e5c87959d4b8abd5b"
server_key_iv: "113240a36d9594a2254389e91d523d25"
server_key_version: "encryptionkey@0"
I20260620 20:27:00.764937 17728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:00.780624 17728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:00.784562 17728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:00.784853 17728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:00.785849 17728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:27:00.787135 17728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:27:00.787238 17728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:00.787387 17728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:27:00.787457 17728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:00.807657 17588 heartbeater.cc:503] Master 127.31.235.190:41693 was elected leader, sending a full tablet report...
I20260620 20:27:00.820542 17728 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:42983
I20260620 20:27:00.820572 17855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:42983 every 8 connection(s)
I20260620 20:27:00.821841 17728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:27:00.824532 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 17728
I20260620 20:27:00.824720 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:27:00.825135 32686 external_mini_cluster.cc:1428] Setting key c77a726249ba157476adbfb761a09771
Jun 20 20:27:00 dist-test-slave-zzwf krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987220, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:27:00.842119 17724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:00.824341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:41101 (local address 127.31.235.190:41693)
0620 20:27:00.824548 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:00.824554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:00.825873 (+ 1319us) server_negotiation.cc:407] Connection header received
0620 20:27:00.827683 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:00.827689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:00.827774 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:00.827883 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:00.829307 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.830107 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.831590 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.831982 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.832110 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:00.835387 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:00.835391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:00.835395 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:00.835426 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:00.839054 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:00.839836 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:00.839842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:00.839845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:00.839994 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:00.840442 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:00.840447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:00.840450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:00.840725 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:00.840899 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:00.841570 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 20:27:00.841800 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:27:00.843014 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:00.824904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:41693 (local address 127.31.235.131:41101)
0620 20:27:00.825710 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:00.825758 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:00.827437 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:00.828119 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:00.828128 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:00.828615 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:00.829139 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:00.829155 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.830260 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.830265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:00.831443 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:00.831455 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.832189 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.832196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:00.832426 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:00.833008 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:27:00.833028 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:00.835195 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:00.839187 (+ 3992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:00.839192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:00.839203 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:00.839713 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:00.840110 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:00.840114 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:00.840117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:00.840322 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:00.840861 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:00.840870 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:00.841185 (+ 315us) client_negotiation.cc:769] Sending connection context
0620 20:27:00.841457 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 20:27:00.841968 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":326,"threads_started":1}
I20260620 20:27:00.844244 17856 heartbeater.cc:347] Connected to a master server at 127.31.235.190:41693
I20260620 20:27:00.844556 17856 heartbeater.cc:464] Registering TS with master...
I20260620 20:27:00.845249 17856 heartbeater.cc:511] Master 127.31.235.190:41693 requested a full tablet report, sending...
I20260620 20:27:00.846614 17399 ts_manager.cc:194] Registered new tserver with Master: ba58824857444423a9a15fa0651ad228 (127.31.235.131:42983)
I20260620 20:27:00.847396 17399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:41101
I20260620 20:27:00.852231 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:27:00.869937 17724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:00.856745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54094 (local address 127.31.235.190:41693)
0620 20:27:00.856963 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:00.856971 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:00.857317 (+ 346us) server_negotiation.cc:407] Connection header received
0620 20:27:00.857442 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:00.857446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:00.857564 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:00.857640 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:00.859111 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.859873 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.861696 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.862218 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.862335 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:00.864759 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:00.864763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:00.864767 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:00.864810 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:00.867363 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:00.868032 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:00.868036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:00.868039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:00.868185 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:00.868513 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:00.868516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:00.868518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:00.868835 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:00.869090 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:00.869324 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 20:27:00.869579 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:27:00.874557 17399 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:27:00.881479 17399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:27:00.905478 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:00.894993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44902 (local address 127.31.235.130:44209)
0620 20:27:00.895940 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:00.895954 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:00.896762 (+ 808us) server_negotiation.cc:407] Connection header received
0620 20:27:00.896886 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:00.896898 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:00.897179 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:00.897351 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:00.898113 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.900018 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.902878 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.904254 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.904435 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:00.904606 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:00.904979 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 20:27:00.905146 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":1753,"threads_started":1}
I20260620 20:27:00.905555 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:00.894732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:44209 (local address 127.0.0.1:44902)
0620 20:27:00.896324 (+ 1592us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:00.896373 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:00.896555 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:00.897381 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:00.897389 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:00.897419 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:00.897862 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:00.897875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.900180 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.900189 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:00.902698 (+ 2509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:00.902715 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.904475 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.904481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:00.904707 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:00.904771 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 20:27:00.904934 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 20:27:00.905128 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1295,"thread_start_us":1435,"threads_started":1}
I20260620 20:27:00.906044 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:00.895531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:41123 (local address 127.0.0.1:54760)
0620 20:27:00.896469 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:00.896489 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:00.896611 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:00.898133 (+ 1522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:00.898142 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:00.898182 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:00.898797 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:00.898813 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.901232 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.901237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:00.903334 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:00.903361 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.905148 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.905154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:00.905334 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:00.905367 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 20:27:00.905440 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 20:27:00.905595 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"mutex_wait_us":23,"spinlock_wait_cycles":2816,"thread_start_us":789,"threads_started":1}
I20260620 20:27:00.906991 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:00.896119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.31.235.129:41123)
0620 20:27:00.897452 (+ 1333us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:00.897463 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:00.897519 (+ 56us) server_negotiation.cc:407] Connection header received
0620 20:27:00.897618 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:00.897626 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:00.897885 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:00.899016 (+ 1131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:00.899324 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.901069 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.903509 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.904919 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.906154 (+ 1235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:00.906309 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:00.906453 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 20:27:00.906636 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1154,"thread_start_us":947,"threads_started":1}
I20260620 20:27:00.908632 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:00.897032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:42983 (local address 127.0.0.1:38850)
0620 20:27:00.899210 (+ 2178us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:00.899226 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:00.899325 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:00.900567 (+ 1242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:00.900572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:00.900597 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:00.901000 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:00.901009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.903127 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.903132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:00.904683 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:00.904695 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.907824 (+ 3129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.907830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:00.908006 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:00.908045 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 20:27:00.908138 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 20:27:00.908263 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2070,"mutex_wait_us":6,"thread_start_us":1936,"threads_started":1}
I20260620 20:27:00.908659 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:00.897615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38850 (local address 127.31.235.131:42983)
0620 20:27:00.899879 (+ 2264us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:00.899886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:00.899913 (+ 27us) server_negotiation.cc:407] Connection header received
0620 20:27:00.899998 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:00.900004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:00.900361 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:00.900573 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:00.901166 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.902992 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.904842 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:00.907600 (+ 2758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:00.907824 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:00.907978 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:00.908180 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 20:27:00.908348 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2115,"thread_start_us":1962,"threads_started":1}
I20260620 20:27:00.909822 17657 tablet_service.cc:1467] Processing CreateTablet for tablet d562621be5394b7898b26c367d0601b2 (DEFAULT_TABLE table=test-table [id=042d4e16d0364c0fa9a776a19173ee38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:27:00.910421 17523 tablet_service.cc:1467] Processing CreateTablet for tablet d562621be5394b7898b26c367d0601b2 (DEFAULT_TABLE table=test-table [id=042d4e16d0364c0fa9a776a19173ee38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:27:00.911340 17657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d562621be5394b7898b26c367d0601b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:00.911765 17523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d562621be5394b7898b26c367d0601b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:00.911942 17791 tablet_service.cc:1467] Processing CreateTablet for tablet d562621be5394b7898b26c367d0601b2 (DEFAULT_TABLE table=test-table [id=042d4e16d0364c0fa9a776a19173ee38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:27:00.913096 17791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d562621be5394b7898b26c367d0601b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:00.925995 17874 tablet_bootstrap.cc:492] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228: Bootstrap starting.
I20260620 20:27:00.928660 17874 tablet_bootstrap.cc:654] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:00.930055 17874 log.cc:824] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:00.930244 17876 tablet_bootstrap.cc:492] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c: Bootstrap starting.
I20260620 20:27:00.930382 17875 tablet_bootstrap.cc:492] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea: Bootstrap starting.
I20260620 20:27:00.933209 17874 tablet_bootstrap.cc:492] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228: No bootstrap required, opened a new log
I20260620 20:27:00.933478 17874 ts_tablet_manager.cc:1397] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 20:27:00.933931 17876 tablet_bootstrap.cc:654] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:00.933969 17875 tablet_bootstrap.cc:654] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:00.935174 17875 log.cc:824] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:00.935225 17876 log.cc:824] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:00.938272 17876 tablet_bootstrap.cc:492] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c: No bootstrap required, opened a new log
I20260620 20:27:00.938395 17875 tablet_bootstrap.cc:492] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea: No bootstrap required, opened a new log
I20260620 20:27:00.938586 17876 ts_tablet_manager.cc:1397] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 20:27:00.938639 17875 ts_tablet_manager.cc:1397] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 20:27:00.944187 17874 raft_consensus.cc:348] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5b4915cac741838e7c8d211172794c" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 44209 } } peers { permanent_uuid: "ba58824857444423a9a15fa0651ad228" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42983 } } peers { permanent_uuid: "9c99da3b4ed7410f8e3245a5162387ea" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41123 } }
I20260620 20:27:00.944660 17874 raft_consensus.cc:374] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:00.944748 17874 raft_consensus.cc:729] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba58824857444423a9a15fa0651ad228, State: Initialized, Role: FOLLOWER
I20260620 20:27:00.945482 17874 consensus_queue.cc:260] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5b4915cac741838e7c8d211172794c" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 44209 } } peers { permanent_uuid: "ba58824857444423a9a15fa0651ad228" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42983 } } peers { permanent_uuid: "9c99da3b4ed7410f8e3245a5162387ea" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41123 } }
I20260620 20:27:00.946096 17876 raft_consensus.cc:348] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5b4915cac741838e7c8d211172794c" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 44209 } } peers { permanent_uuid: "ba58824857444423a9a15fa0651ad228" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42983 } } peers { permanent_uuid: "9c99da3b4ed7410f8e3245a5162387ea" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41123 } }
I20260620 20:27:00.946524 17876 raft_consensus.cc:374] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:00.946627 17876 raft_consensus.cc:729] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba5b4915cac741838e7c8d211172794c, State: Initialized, Role: FOLLOWER
I20260620 20:27:00.946799 17856 heartbeater.cc:503] Master 127.31.235.190:41693 was elected leader, sending a full tablet report...
I20260620 20:27:00.947295 17876 consensus_queue.cc:260] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5b4915cac741838e7c8d211172794c" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 44209 } } peers { permanent_uuid: "ba58824857444423a9a15fa0651ad228" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42983 } } peers { permanent_uuid: "9c99da3b4ed7410f8e3245a5162387ea" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41123 } }
I20260620 20:27:00.947470 17874 ts_tablet_manager.cc:1428] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260620 20:27:00.948294 17722 heartbeater.cc:503] Master 127.31.235.190:41693 was elected leader, sending a full tablet report...
I20260620 20:27:00.948879 17876 ts_tablet_manager.cc:1428] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260620 20:27:00.950796 17875 raft_consensus.cc:348] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5b4915cac741838e7c8d211172794c" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 44209 } } peers { permanent_uuid: "ba58824857444423a9a15fa0651ad228" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42983 } } peers { permanent_uuid: "9c99da3b4ed7410f8e3245a5162387ea" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41123 } }
I20260620 20:27:00.951246 17875 raft_consensus.cc:374] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:00.951388 17875 raft_consensus.cc:729] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c99da3b4ed7410f8e3245a5162387ea, State: Initialized, Role: FOLLOWER
I20260620 20:27:00.951949 17875 consensus_queue.cc:260] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5b4915cac741838e7c8d211172794c" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 44209 } } peers { permanent_uuid: "ba58824857444423a9a15fa0651ad228" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42983 } } peers { permanent_uuid: "9c99da3b4ed7410f8e3245a5162387ea" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41123 } }
I20260620 20:27:00.953724 17875 ts_tablet_manager.cc:1428] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260620 20:27:01.025935 17589 tablet.cc:2369] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:27:01.068300 17723 tablet.cc:2369] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:27:01.073539 17857 tablet.cc:2369] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:27:01.209899 17882 raft_consensus.cc:482] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:27:01.210220 17882 raft_consensus.cc:504] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5b4915cac741838e7c8d211172794c" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 44209 } } peers { permanent_uuid: "ba58824857444423a9a15fa0651ad228" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42983 } } peers { permanent_uuid: "9c99da3b4ed7410f8e3245a5162387ea" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41123 } }
I20260620 20:27:01.211927 17882 leader_election.cc:288] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba5b4915cac741838e7c8d211172794c (127.31.235.130:44209), ba58824857444423a9a15fa0651ad228 (127.31.235.131:42983)
I20260620 20:27:01.218662 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:01.212317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:42733 (local address 127.31.235.130:44209)
0620 20:27:01.212519 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:01.212524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:01.212991 (+ 467us) server_negotiation.cc:407] Connection header received
0620 20:27:01.213226 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:01.213231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:01.213312 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:01.213424 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:01.214130 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:01.214842 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:01.216729 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:01.217714 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:01.217885 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:01.217970 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:01.218279 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 20:27:01.218397 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:27:01.218811 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:01.212224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:44209 (local address 127.31.235.129:42733)
0620 20:27:01.212882 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:01.212904 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:01.213059 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:01.213599 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:01.213604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:01.213632 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:01.213985 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:01.213993 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:01.214986 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:01.214994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:01.216588 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:01.216600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:01.217905 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:01.217911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:01.218072 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:01.218139 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 20:27:01.218244 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 20:27:01.218381 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":3,"thread_start_us":345,"threads_started":1}
I20260620 20:27:01.219027 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:01.213282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:51001 (local address 127.31.235.131:42983)
0620 20:27:01.213485 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:01.213512 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:01.213809 (+ 297us) server_negotiation.cc:407] Connection header received
0620 20:27:01.214011 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:01.214016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:01.214080 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:01.214177 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:01.214727 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:01.215449 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:01.217235 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:01.218108 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:01.218285 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:01.218362 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:01.218641 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 20:27:01.218761 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:27:01.219184 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:01.213036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:42983 (local address 127.31.235.129:51001)
0620 20:27:01.213687 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:01.213700 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:01.213872 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:01.214191 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:01.214195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:01.214217 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:01.214579 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:01.214587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:01.215590 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:01.215596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:01.217067 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:01.217079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:01.218331 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:01.218337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:01.218486 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:01.218513 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 20:27:01.218616 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 20:27:01.218742 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":22,"thread_start_us":339,"threads_started":1}
I20260620 20:27:01.220216 17811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d562621be5394b7898b26c367d0601b2" candidate_uuid: "9c99da3b4ed7410f8e3245a5162387ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba58824857444423a9a15fa0651ad228" is_pre_election: true
I20260620 20:27:01.220211 17677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d562621be5394b7898b26c367d0601b2" candidate_uuid: "9c99da3b4ed7410f8e3245a5162387ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba5b4915cac741838e7c8d211172794c" is_pre_election: true
I20260620 20:27:01.220796 17677 raft_consensus.cc:2454] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c99da3b4ed7410f8e3245a5162387ea in term 0.
I20260620 20:27:01.220783 17811 raft_consensus.cc:2454] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c99da3b4ed7410f8e3245a5162387ea in term 0.
I20260620 20:27:01.221457 17478 leader_election.cc:302] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c99da3b4ed7410f8e3245a5162387ea, ba58824857444423a9a15fa0651ad228; no voters:
I20260620 20:27:01.221884 17882 raft_consensus.cc:2781] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:27:01.222059 17882 raft_consensus.cc:482] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:27:01.222193 17882 raft_consensus.cc:3037] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:01.224545 17882 raft_consensus.cc:504] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5b4915cac741838e7c8d211172794c" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 44209 } } peers { permanent_uuid: "ba58824857444423a9a15fa0651ad228" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42983 } } peers { permanent_uuid: "9c99da3b4ed7410f8e3245a5162387ea" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41123 } }
I20260620 20:27:01.225134 17882 leader_election.cc:288] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea [CANDIDATE]: Term 1 election: Requested vote from peers ba5b4915cac741838e7c8d211172794c (127.31.235.130:44209), ba58824857444423a9a15fa0651ad228 (127.31.235.131:42983)
I20260620 20:27:01.225687 17677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d562621be5394b7898b26c367d0601b2" candidate_uuid: "9c99da3b4ed7410f8e3245a5162387ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba5b4915cac741838e7c8d211172794c"
I20260620 20:27:01.225742 17811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d562621be5394b7898b26c367d0601b2" candidate_uuid: "9c99da3b4ed7410f8e3245a5162387ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba58824857444423a9a15fa0651ad228"
I20260620 20:27:01.225929 17677 raft_consensus.cc:3037] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:01.225943 17811 raft_consensus.cc:3037] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:01.229112 17677 raft_consensus.cc:2454] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c99da3b4ed7410f8e3245a5162387ea in term 1.
I20260620 20:27:01.229112 17811 raft_consensus.cc:2454] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c99da3b4ed7410f8e3245a5162387ea in term 1.
I20260620 20:27:01.229532 17478 leader_election.cc:302] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c99da3b4ed7410f8e3245a5162387ea, ba58824857444423a9a15fa0651ad228; no voters:
I20260620 20:27:01.229821 17882 raft_consensus.cc:2781] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:01.230316 17882 raft_consensus.cc:686] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea [term 1 LEADER]: Becoming Leader. State: Replica: 9c99da3b4ed7410f8e3245a5162387ea, State: Running, Role: LEADER
I20260620 20:27:01.230782 17882 consensus_queue.cc:237] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5b4915cac741838e7c8d211172794c" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 44209 } } peers { permanent_uuid: "ba58824857444423a9a15fa0651ad228" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42983 } } peers { permanent_uuid: "9c99da3b4ed7410f8e3245a5162387ea" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41123 } }
I20260620 20:27:01.235371 17399 catalog_manager.cc:5495] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c99da3b4ed7410f8e3245a5162387ea (127.31.235.129). New cstate: current_term: 1 leader_uuid: "9c99da3b4ed7410f8e3245a5162387ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba5b4915cac741838e7c8d211172794c" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 44209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba58824857444423a9a15fa0651ad228" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c99da3b4ed7410f8e3245a5162387ea" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41123 } health_report { overall_health: HEALTHY } } }
I20260620 20:27:01.271245 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:01.265871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54768 (local address 127.31.235.129:41123)
0620 20:27:01.266058 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:01.266063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:01.266128 (+ 65us) server_negotiation.cc:407] Connection header received
0620 20:27:01.266324 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:01.266328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:01.266405 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:01.266528 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:27:01.266829 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:01.267566 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:01.269038 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:01.269454 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:01.269638 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:01.270115 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:27:01.270694 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:27:01.270811 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:01.270882 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 20:27:01.270953 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 20:27:01.279225 17811 raft_consensus.cc:1261] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228 [term 1 FOLLOWER]: Refusing update from remote peer 9c99da3b4ed7410f8e3245a5162387ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:27:01.279268 17677 raft_consensus.cc:1261] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c [term 1 FOLLOWER]: Refusing update from remote peer 9c99da3b4ed7410f8e3245a5162387ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:27:01.280181 17885 consensus_queue.cc:1035] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba5b4915cac741838e7c8d211172794c" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 44209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:27:01.280628 17882 consensus_queue.cc:1035] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba58824857444423a9a15fa0651ad228" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 42983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:27:01.290211 17890 mvcc.cc:205] Tried to move back new op lower bound from 7299019658350030848 to 7299019658164830208. Current Snapshot: MvccSnapshot[applied={T|T < 7299019658350030848}]
I20260620 20:27:01.290185 17888 mvcc.cc:205] Tried to move back new op lower bound from 7299019658350030848 to 7299019658164830208. Current Snapshot: MvccSnapshot[applied={T|T < 7299019658350030848}]
I20260620 20:27:01.306545 17399 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:27:01.306846 17399 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:27:01.310748 17399 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15aa01658b844f1c8ab690b85bca3fdb: Sending DeleteTablet for 3 replicas of tablet d562621be5394b7898b26c367d0601b2
I20260620 20:27:01.312153 17523 tablet_service.cc:1514] Processing DeleteTablet for tablet d562621be5394b7898b26c367d0601b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:27:01 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:54760
I20260620 20:27:01.312479 17791 tablet_service.cc:1514] Processing DeleteTablet for tablet d562621be5394b7898b26c367d0601b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:27:01 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:38850
I20260620 20:27:01.312860 17657 tablet_service.cc:1514] Processing DeleteTablet for tablet d562621be5394b7898b26c367d0601b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:27:01 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:44902
I20260620 20:27:01.313055 17898 tablet_replica.cc:330] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea: stopping tablet replica
I20260620 20:27:01.313634 17898 raft_consensus.cc:2229] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:27:01.313825 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 17460
I20260620 20:27:01.314361 17898 raft_consensus.cc:2258] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:27:01.319958 17898 ts_tablet_manager.cc:1905] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:27:01.328462 17898 ts_tablet_manager.cc:1918] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:27:01.328670 17898 log.cc:1196] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/d562621be5394b7898b26c367d0601b2
I20260620 20:27:01.329201 17898 ts_tablet_manager.cc:1939] T d562621be5394b7898b26c367d0601b2 P 9c99da3b4ed7410f8e3245a5162387ea: Deleting consensus metadata
I20260620 20:27:01.331303 17384 catalog_manager.cc:4841] TS 9c99da3b4ed7410f8e3245a5162387ea (127.31.235.129:41123): tablet d562621be5394b7898b26c367d0601b2 (table test-table [id=042d4e16d0364c0fa9a776a19173ee38]) successfully deleted
I20260620 20:27:01.338438 17900 tablet_replica.cc:330] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c: stopping tablet replica
I20260620 20:27:01.339043 17900 raft_consensus.cc:2229] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:27:01.339643 17900 raft_consensus.cc:2258] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:27:01.340448 17899 tablet_replica.cc:330] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228: stopping tablet replica
I20260620 20:27:01.340905 17899 raft_consensus.cc:2229] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:27:01.341346 17899 raft_consensus.cc:2258] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:27:01.343551 17900 ts_tablet_manager.cc:1905] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 20:27:01 dist-test-slave-zzwf krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:27:01.350659 17900 ts_tablet_manager.cc:1918] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:27:01.350816 17900 log.cc:1196] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/d562621be5394b7898b26c367d0601b2
I20260620 20:27:01.351351 17900 ts_tablet_manager.cc:1939] T d562621be5394b7898b26c367d0601b2 P ba5b4915cac741838e7c8d211172794c: Deleting consensus metadata
I20260620 20:27:01.353085 17385 catalog_manager.cc:4841] TS ba5b4915cac741838e7c8d211172794c (127.31.235.130:44209): tablet d562621be5394b7898b26c367d0601b2 (table test-table [id=042d4e16d0364c0fa9a776a19173ee38]) successfully deleted
I20260620 20:27:01.355054 17899 ts_tablet_manager.cc:1905] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:27:01.358487 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:01.318013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54770 (local address 127.31.235.129:41123)
0620 20:27:01.318234 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:01.318242 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:01.338117 (+ 19875us) server_negotiation.cc:407] Connection header received
0620 20:27:01.338441 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:01.338447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:01.338574 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:01.338652 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:01.340919 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:01.342212 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:01.344350 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:01.345098 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:01.345203 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:01.350812 (+ 5609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:01.350829 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:01.350847 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:01.350913 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:01.355267 (+ 4354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:01.356131 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:01.356141 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:01.356145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:01.356324 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:01.356733 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:01.356737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:01.356740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:01.357188 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:01.357549 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:01.357833 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 20:27:01.358108 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:27:01.359277 17583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:01.362901 17899 ts_tablet_manager.cc:1918] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:27:01.363108 17899 log.cc:1196] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/d562621be5394b7898b26c367d0601b2
I20260620 20:27:01.363602 17899 ts_tablet_manager.cc:1939] T d562621be5394b7898b26c367d0601b2 P ba58824857444423a9a15fa0651ad228: Deleting consensus metadata
I20260620 20:27:01.365187 17385 catalog_manager.cc:4841] TS ba58824857444423a9a15fa0651ad228 (127.31.235.131:42983): tablet d562621be5394b7898b26c367d0601b2 (table test-table [id=042d4e16d0364c0fa9a776a19173ee38]) successfully deleted
I20260620 20:27:01.456760 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 17460
I20260620 20:27:01.475160 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 17594
Jun 20 20:27:01 dist-test-slave-zzwf krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.130@KRBTEST.COM
I20260620 20:27:01.490553 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:01.475993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44904 (local address 127.31.235.130:44209)
0620 20:27:01.476195 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:01.476201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:01.476221 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:27:01.476372 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:01.476377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:01.476469 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:01.476576 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:01.477833 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:01.478550 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:01.479763 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:01.480190 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:01.480318 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:01.484565 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:01.484580 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:01.484596 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:01.484631 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:01.487491 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:01.488177 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:01.488185 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:01.488187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:01.488311 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:01.488692 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:01.488698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:01.488701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:01.489073 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:01.489370 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:01.489964 (+ 594us) server_negotiation.cc:299] Negotiation successful
0620 20:27:01.490186 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:27:01.491245 17717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:01.582317 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 17594
I20260620 20:27:01.599009 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 17728
Jun 20 20:27:01 dist-test-slave-zzwf krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.131@KRBTEST.COM
I20260620 20:27:01.613919 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:01.599782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38854 (local address 127.31.235.131:42983)
0620 20:27:01.599963 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:01.599968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:01.599986 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:27:01.600067 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:01.600072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:01.600133 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:01.600214 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:01.601483 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:01.602250 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:01.603639 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:01.604023 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:01.604115 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:01.608526 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:01.608540 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:01.608556 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:01.608593 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:01.611366 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:01.612065 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:01.612072 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:01.612074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:01.612194 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:01.612544 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:01.612547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:01.612549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:01.612878 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:01.613102 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:01.613366 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 20:27:01.613658 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:27:01.614753 17851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:01.704661 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 17728
I20260620 20:27:01.722119 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 17367
I20260620 20:27:01.723640 17429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:01.833835 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 17367
2026-06-20T20:27:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4459 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T20:27:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:01Z Disabled control of system clock
I20260620 20:27:01.912254 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:45741
--webserver_interface=127.31.235.190
--webserver_port=0
--builtin_ntp_servers=127.31.235.148:36547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:45741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:27:02.255348 17909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:02.255708 17909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:02.255836 17909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:02.263329 17909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:02.263450 17909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:02.263536 17909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:02.263628 17909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:02.263698 17909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:02.273775 17909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:36547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:45741
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:45741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:02.275170 17909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:02.277030 17909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:02.288980 17914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:02.289011 17915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:02.289036 17917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:02.289870 17909 server_base.cc:1034] running on GCE node
I20260620 20:27:02.290676 17909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:02.292110 17909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:02.293334 17909 hybrid_clock.cc:648] HybridClock initialized: now 1781987222293269 us; error 54 us; skew 500 ppm
I20260620 20:27:02.293727 17909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:27:02.300235 17909 webserver.cc:460] Webserver started at http://127.31.235.190:45777/ using document root <none> and password file <none>
I20260620 20:27:02.300971 17909 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:02.301105 17909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:02.301424 17909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:02.304358 17909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "8ee8828ad1444664bc4cb61804e46112"
format_stamp: "Formatted at 2026-06-20 20:27:02 on dist-test-slave-zzwf"
server_key: "368828d005cb423cd45c400ba389fd83"
server_key_iv: "fc03c82e53e0f400a5749c7a9450540d"
server_key_version: "encryptionkey@0"
I20260620 20:27:02.305186 17909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "8ee8828ad1444664bc4cb61804e46112"
format_stamp: "Formatted at 2026-06-20 20:27:02 on dist-test-slave-zzwf"
server_key: "368828d005cb423cd45c400ba389fd83"
server_key_iv: "fc03c82e53e0f400a5749c7a9450540d"
server_key_version: "encryptionkey@0"
I20260620 20:27:02.310552 17909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 20:27:02.314062 17923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:02.315433 17909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 20:27:02.315616 17909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "8ee8828ad1444664bc4cb61804e46112"
format_stamp: "Formatted at 2026-06-20 20:27:02 on dist-test-slave-zzwf"
server_key: "368828d005cb423cd45c400ba389fd83"
server_key_iv: "fc03c82e53e0f400a5749c7a9450540d"
server_key_version: "encryptionkey@0"
I20260620 20:27:02.315830 17909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:02.349776 17909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:02.350670 17909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:02.350970 17909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:02.372730 17909 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:45741
I20260620 20:27:02.372741 17974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:45741 every 8 connection(s)
I20260620 20:27:02.374639 17909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:02.377719 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 17909
I20260620 20:27:02.377878 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:02.378456 32686 external_mini_cluster.cc:1428] Setting key 1ca202fa2fe16816fe766a2189a3d7a9
I20260620 20:27:02.379745 17975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:02.396946 17975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112: Bootstrap starting.
I20260620 20:27:02.397166 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:02.380958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60092 (local address 127.31.235.190:45741)
0620 20:27:02.381920 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:02.381930 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:02.381971 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:27:02.385697 (+ 3726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:02.385722 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:02.386212 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:02.386628 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:02.387199 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:02.388533 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:02.390363 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:02.391412 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:02.391590 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:02.393062 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:02.393072 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:02.393091 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:02.393907 (+ 816us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:02.394039 (+ 132us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:02.394070 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:02.394186 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:02.395506 (+ 1320us) server_negotiation.cc:299] Negotiation successful
0620 20:27:02.395830 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":343,"threads_started":1}
I20260620 20:27:02.401248 17975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:02.402696 17975 log.cc:824] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:02.406493 17975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112: No bootstrap required, opened a new log
I20260620 20:27:02.419696 17975 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee8828ad1444664bc4cb61804e46112" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 45741 } }
I20260620 20:27:02.420246 17975 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:02.420367 17975 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee8828ad1444664bc4cb61804e46112, State: Initialized, Role: FOLLOWER
I20260620 20:27:02.421087 17975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee8828ad1444664bc4cb61804e46112" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 45741 } }
I20260620 20:27:02.421377 17975 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:02.421553 17975 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:02.421705 17975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:02.424077 17975 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee8828ad1444664bc4cb61804e46112" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 45741 } }
I20260620 20:27:02.424539 17975 leader_election.cc:302] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ee8828ad1444664bc4cb61804e46112; no voters:
I20260620 20:27:02.425045 17975 leader_election.cc:288] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:02.425201 17980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:02.425953 17980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee8828ad1444664bc4cb61804e46112, State: Running, Role: LEADER
I20260620 20:27:02.426448 17980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee8828ad1444664bc4cb61804e46112" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 45741 } }
I20260620 20:27:02.427306 17975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:02.429248 17981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ee8828ad1444664bc4cb61804e46112" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee8828ad1444664bc4cb61804e46112" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 45741 } } }
I20260620 20:27:02.429869 17981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:02.429275 17982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ee8828ad1444664bc4cb61804e46112. Latest consensus state: current_term: 1 leader_uuid: "8ee8828ad1444664bc4cb61804e46112" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee8828ad1444664bc4cb61804e46112" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 45741 } } }
I20260620 20:27:02.431650 17982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:02.434070 17985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:02.439293 17985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:02.447813 17985 catalog_manager.cc:1347] Generated new cluster ID: a8d0d458c35e46ba993648351ba3fc4a
I20260620 20:27:02.447935 17985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:02.456516 17985 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:02.457840 17985 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:02.464850 17985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112: Generated new TSK 0
I20260620 20:27:02.465860 17985 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:02.470961 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:45741
--builtin_ntp_servers=127.31.235.148:36547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:27:02.806492 17999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:02.806766 17999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:02.806929 17999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:02.814121 17999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:02.814221 17999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:02.814427 17999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:27:02.823634 17999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:36547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:45741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:02.824945 17999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:02.826820 17999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:02.840416 18007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:02.840416 18004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:02.840478 18005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:02.843104 17999 server_base.cc:1034] running on GCE node
I20260620 20:27:02.843758 17999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:02.844726 17999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:02.845937 17999 hybrid_clock.cc:648] HybridClock initialized: now 1781987222845891 us; error 42 us; skew 500 ppm
I20260620 20:27:02.846289 17999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:27:02.853193 17999 webserver.cc:460] Webserver started at http://127.31.235.129:36667/ using document root <none> and password file <none>
I20260620 20:27:02.854051 17999 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:02.854182 17999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:02.854516 17999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:02.857353 17999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "55e49a4bf8094a66ad05f5fe4b2a01f4"
format_stamp: "Formatted at 2026-06-20 20:27:02 on dist-test-slave-zzwf"
server_key: "359a7fb61d70bc183efb935766cb395e"
server_key_iv: "ca91d94486a772540148aec4fcdc175e"
server_key_version: "encryptionkey@0"
I20260620 20:27:02.858140 17999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "55e49a4bf8094a66ad05f5fe4b2a01f4"
format_stamp: "Formatted at 2026-06-20 20:27:02 on dist-test-slave-zzwf"
server_key: "359a7fb61d70bc183efb935766cb395e"
server_key_iv: "ca91d94486a772540148aec4fcdc175e"
server_key_version: "encryptionkey@0"
I20260620 20:27:02.863181 17999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:27:02.866644 18013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:02.868207 17999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:27:02.868415 17999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "55e49a4bf8094a66ad05f5fe4b2a01f4"
format_stamp: "Formatted at 2026-06-20 20:27:02 on dist-test-slave-zzwf"
server_key: "359a7fb61d70bc183efb935766cb395e"
server_key_iv: "ca91d94486a772540148aec4fcdc175e"
server_key_version: "encryptionkey@0"
I20260620 20:27:02.868657 17999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:02.884757 17999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:02.885720 17999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:02.886027 17999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:02.887036 17999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:27:02.888568 17999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:27:02.888691 17999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:02.888829 17999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:27:02.888918 17999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:02.924345 17999 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:39663
I20260620 20:27:02.924396 18125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:39663 every 8 connection(s)
I20260620 20:27:02.925762 17999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:27:02.928512 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 17999
I20260620 20:27:02.928702 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:27:02.929210 32686 external_mini_cluster.cc:1428] Setting key 1fb0559c375a963214d1b97d4ce11374
I20260620 20:27:02.932284 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:45741
--builtin_ntp_servers=127.31.235.148:36547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:27:02.941531 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:02.928498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:38937 (local address 127.31.235.190:45741)
0620 20:27:02.929068 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:02.929074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:02.930422 (+ 1348us) server_negotiation.cc:407] Connection header received
0620 20:27:02.933063 (+ 2641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:02.933068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:02.933140 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:02.933252 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:02.934760 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:02.935513 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:02.938133 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:02.938593 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:02.938711 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:02.940494 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:02.940500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:02.940503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:02.940527 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:02.940586 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:02.940601 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:02.940679 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:02.941041 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 20:27:02.941145 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":288,"threads_started":1}
I20260620 20:27:02.942601 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:02.929164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:45741 (local address 127.31.235.129:38937)
0620 20:27:02.930273 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:02.930310 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:02.932748 (+ 2438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:02.933584 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:02.933599 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:02.933730 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:02.934561 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:02.934588 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:02.935695 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:02.935703 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:02.937968 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:02.937988 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:02.938840 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:02.938850 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:02.939195 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:02.940035 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:27:02.940041 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:02.940100 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:27:02.940214 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:02.940707 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:02.940721 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:02.940733 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 20:27:02.940995 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 20:27:02.941420 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":331,"threads_started":1}
I20260620 20:27:02.944533 18126 heartbeater.cc:347] Connected to a master server at 127.31.235.190:45741
I20260620 20:27:02.944852 18126 heartbeater.cc:464] Registering TS with master...
I20260620 20:27:02.945698 18126 heartbeater.cc:511] Master 127.31.235.190:45741 requested a full tablet report, sending...
I20260620 20:27:02.948063 17940 ts_manager.cc:194] Registered new tserver with Master: 55e49a4bf8094a66ad05f5fe4b2a01f4 (127.31.235.129:39663)
I20260620 20:27:02.950546 17940 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.235.129:38937
W20260620 20:27:03.273051 18130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:03.273348 18130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:03.273437 18130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:03.280853 18130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:03.280959 18130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:03.281068 18130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:27:03.290462 18130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:36547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:45741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:03.291822 18130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:03.293645 18130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:03.306825 18135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:03.306802 18138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:03.307085 18136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:03.308085 18130 server_base.cc:1034] running on GCE node
I20260620 20:27:03.308706 18130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:03.309836 18130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:03.311117 18130 hybrid_clock.cc:648] HybridClock initialized: now 1781987223311059 us; error 65 us; skew 500 ppm
I20260620 20:27:03.311480 18130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:27:03.314092 18130 webserver.cc:460] Webserver started at http://127.31.235.130:39169/ using document root <none> and password file <none>
I20260620 20:27:03.314841 18130 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:03.314970 18130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:03.315295 18130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:03.318293 18130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "502b8ea254a747059f81a400d3d57575"
format_stamp: "Formatted at 2026-06-20 20:27:03 on dist-test-slave-zzwf"
server_key: "fc59a7920035db1e590796733e1c0a07"
server_key_iv: "c1939c76ed237020170f1d02e007111b"
server_key_version: "encryptionkey@0"
I20260620 20:27:03.319099 18130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "502b8ea254a747059f81a400d3d57575"
format_stamp: "Formatted at 2026-06-20 20:27:03 on dist-test-slave-zzwf"
server_key: "fc59a7920035db1e590796733e1c0a07"
server_key_iv: "c1939c76ed237020170f1d02e007111b"
server_key_version: "encryptionkey@0"
I20260620 20:27:03.324692 18130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:27:03.328157 18144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:03.329646 18130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 20:27:03.329813 18130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "502b8ea254a747059f81a400d3d57575"
format_stamp: "Formatted at 2026-06-20 20:27:03 on dist-test-slave-zzwf"
server_key: "fc59a7920035db1e590796733e1c0a07"
server_key_iv: "c1939c76ed237020170f1d02e007111b"
server_key_version: "encryptionkey@0"
I20260620 20:27:03.330040 18130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:03.346379 18130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:03.347225 18130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:03.347529 18130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:03.348524 18130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:27:03.350030 18130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:27:03.350147 18130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:03.350282 18130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:27:03.350378 18130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:03.386365 18130 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:39007
I20260620 20:27:03.386420 18256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:39007 every 8 connection(s)
I20260620 20:27:03.387938 18130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:27:03.394191 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 18130
I20260620 20:27:03.394368 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:27:03.394891 32686 external_mini_cluster.cc:1428] Setting key d6738db82a1ff134732dbc591436202d
I20260620 20:27:03.397742 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:45741
--builtin_ntp_servers=127.31.235.148:36547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:27:03.402029 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:03.391009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:45523 (local address 127.31.235.190:45741)
0620 20:27:03.391235 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:03.391242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:03.392914 (+ 1672us) server_negotiation.cc:407] Connection header received
0620 20:27:03.394959 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:03.394966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:03.395068 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:03.395193 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:03.396429 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.397337 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.399078 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.399559 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.399717 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:03.401154 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:03.401160 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:03.401163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:03.401180 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:03.401253 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:03.401267 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:03.401367 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:03.401633 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 20:27:03.401733 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 20:27:03.403044 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:03.391604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:45741 (local address 127.31.235.130:45523)
0620 20:27:03.392707 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:03.392768 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:03.394642 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:03.395484 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:03.395497 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:03.395622 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:03.396252 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:03.396271 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.397537 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.397543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:03.398918 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:03.398931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.399811 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.399819 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:03.400060 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:03.400829 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:27:03.400832 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:03.400872 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:27:03.400948 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:03.401365 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:03.401377 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:03.401387 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 20:27:03.401612 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 20:27:03.402012 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":363,"threads_started":1}
I20260620 20:27:03.404412 18257 heartbeater.cc:347] Connected to a master server at 127.31.235.190:45741
I20260620 20:27:03.404721 18257 heartbeater.cc:464] Registering TS with master...
I20260620 20:27:03.405455 18257 heartbeater.cc:511] Master 127.31.235.190:45741 requested a full tablet report, sending...
I20260620 20:27:03.406819 17940 ts_manager.cc:194] Registered new tserver with Master: 502b8ea254a747059f81a400d3d57575 (127.31.235.130:39007)
I20260620 20:27:03.407626 17940 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.235.130:45523
W20260620 20:27:03.734498 18260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:03.734812 18260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:03.734992 18260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:03.742543 18260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:03.742646 18260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:03.742808 18260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:27:03.752058 18260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:36547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:45741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:03.753364 18260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:03.755115 18260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:03.771119 18265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:03.771638 18268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:03.771122 18266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:03.772603 18260 server_base.cc:1034] running on GCE node
I20260620 20:27:03.773384 18260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:03.774444 18260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:03.775678 18260 hybrid_clock.cc:648] HybridClock initialized: now 1781987223775616 us; error 59 us; skew 500 ppm
I20260620 20:27:03.776036 18260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:27:03.778509 18260 webserver.cc:460] Webserver started at http://127.31.235.131:46815/ using document root <none> and password file <none>
I20260620 20:27:03.779188 18260 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:03.779307 18260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:03.779621 18260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:03.782552 18260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "3c248de25fbe42308ed8c33b28886256"
format_stamp: "Formatted at 2026-06-20 20:27:03 on dist-test-slave-zzwf"
server_key: "a5a477a4044eb3bd6f91922e9c80c893"
server_key_iv: "3b3c5bacbc54290860d5dd357f7f3566"
server_key_version: "encryptionkey@0"
I20260620 20:27:03.783329 18260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "3c248de25fbe42308ed8c33b28886256"
format_stamp: "Formatted at 2026-06-20 20:27:03 on dist-test-slave-zzwf"
server_key: "a5a477a4044eb3bd6f91922e9c80c893"
server_key_iv: "3b3c5bacbc54290860d5dd357f7f3566"
server_key_version: "encryptionkey@0"
I20260620 20:27:03.788484 18260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:27:03.791973 18274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:03.793362 18260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 20:27:03.793540 18260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "3c248de25fbe42308ed8c33b28886256"
format_stamp: "Formatted at 2026-06-20 20:27:03 on dist-test-slave-zzwf"
server_key: "a5a477a4044eb3bd6f91922e9c80c893"
server_key_iv: "3b3c5bacbc54290860d5dd357f7f3566"
server_key_version: "encryptionkey@0"
I20260620 20:27:03.793753 18260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:03.813194 18260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:03.814199 18260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:03.814604 18260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:03.815861 18260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:27:03.817456 18260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:27:03.817610 18260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 20:27:03.817727 18260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:27:03.817814 18260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:03.857388 18260 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:41247
I20260620 20:27:03.857411 18386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:41247 every 8 connection(s)
I20260620 20:27:03.858899 18260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:27:03.866340 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 18260
I20260620 20:27:03.866560 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:27:03.867204 32686 external_mini_cluster.cc:1428] Setting key 8f8e5d8e2e64999745bbb804b6aae2b9
I20260620 20:27:03.872872 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:03.861636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:45803 (local address 127.31.235.190:45741)
0620 20:27:03.861821 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:03.861826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:03.863479 (+ 1653us) server_negotiation.cc:407] Connection header received
0620 20:27:03.865243 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:03.865250 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:03.865354 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:03.865465 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:03.866494 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.867234 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.869241 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.869951 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.870087 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:03.871820 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:03.871829 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:03.871837 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:03.871863 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:03.871915 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:03.871933 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:03.872042 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:03.872238 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 20:27:03.872405 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:27:03.873838 18389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:03.862350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:45741 (local address 127.31.235.131:45803)
0620 20:27:03.863309 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:03.863353 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:03.864979 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:03.865696 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:03.865704 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:03.865796 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:03.866330 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:03.866346 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.867400 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.867407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:03.869055 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:03.869071 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.870356 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.870364 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:03.870606 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:03.871464 (+ 858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:27:03.871467 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:03.871507 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:27:03.871594 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:03.872030 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:03.872038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:03.872043 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 20:27:03.872185 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 20:27:03.872622 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":317,"threads_started":1}
I20260620 20:27:03.875319 18387 heartbeater.cc:347] Connected to a master server at 127.31.235.190:45741
I20260620 20:27:03.875653 18387 heartbeater.cc:464] Registering TS with master...
I20260620 20:27:03.876415 18387 heartbeater.cc:511] Master 127.31.235.190:45741 requested a full tablet report, sending...
I20260620 20:27:03.877875 17940 ts_manager.cc:194] Registered new tserver with Master: 3c248de25fbe42308ed8c33b28886256 (127.31.235.131:41247)
I20260620 20:27:03.878698 17940 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.235.131:45803
I20260620 20:27:03.882763 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:27:03.894254 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:03.887038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60096 (local address 127.31.235.190:45741)
0620 20:27:03.887275 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:03.887286 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:03.887781 (+ 495us) server_negotiation.cc:407] Connection header received
0620 20:27:03.887981 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:03.887985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:03.888061 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:03.888145 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:03.888880 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.889716 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.891903 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.892327 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.892427 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:03.893581 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:03.893585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:03.893589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:03.893607 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:03.893638 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:03.893650 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:03.893721 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:03.893838 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 20:27:03.893912 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:27:03.898168 17940 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:27:03.905093 17940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:27:03.942605 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:03.918201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:39663 (local address 127.0.0.1:37000)
0620 20:27:03.918994 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:03.919030 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:03.934378 (+ 15348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:03.935238 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:03.935246 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:03.935292 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:03.935752 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:03.935764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.937866 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.937872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:03.940156 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:03.940168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.941628 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.941635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:03.941839 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:03.941891 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 20:27:03.942032 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 20:27:03.942213 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":15,"thread_start_us":346,"threads_started":1}
I20260620 20:27:03.942619 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:03.918435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37000 (local address 127.31.235.129:39663)
0620 20:27:03.931092 (+ 12657us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:03.931105 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:03.934594 (+ 3489us) server_negotiation.cc:407] Connection header received
0620 20:27:03.934697 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:03.934705 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:03.934967 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:03.935170 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:03.935940 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.937691 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.940325 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.941408 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.941632 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:03.941870 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:03.942095 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 20:27:03.942260 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12499,"thread_start_us":12372,"threads_started":1}
I20260620 20:27:03.943856 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:03.919980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37346 (local address 127.31.235.131:41247)
0620 20:27:03.932627 (+ 12647us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:03.932635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:03.934873 (+ 2238us) server_negotiation.cc:407] Connection header received
0620 20:27:03.934975 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:03.934980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:03.935185 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:03.935370 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:03.936514 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.938159 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.941148 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.942402 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.942593 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:03.942795 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:03.943315 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 20:27:03.943473 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12485,"thread_start_us":12655,"threads_started":1}
I20260620 20:27:03.944541 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:03.919437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41482 (local address 127.31.235.130:39007)
0620 20:27:03.930681 (+ 11244us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:03.930691 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:03.934311 (+ 3620us) server_negotiation.cc:407] Connection header received
0620 20:27:03.935144 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:03.935155 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:03.935362 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:03.935517 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:03.936526 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.938137 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.941122 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.942544 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.943955 (+ 1411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:03.944140 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:03.944205 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 20:27:03.944315 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11048,"thread_start_us":10935,"threads_started":1}
I20260620 20:27:03.943791 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:03.919443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:41247 (local address 127.0.0.1:37346)
0620 20:27:03.934569 (+ 15126us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:03.934587 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:03.934705 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:03.935650 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:03.935657 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:03.935686 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:03.936269 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:03.936284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.938340 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.938347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:03.940901 (+ 2554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:03.940917 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.942927 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.942933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:03.943143 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:03.943169 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 20:27:03.943282 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 20:27:03.943400 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14996,"mutex_wait_us":124,"spinlock_wait_cycles":1280,"thread_start_us":14394,"threads_started":1}
I20260620 20:27:03.944893 18404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:03.918784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:39007 (local address 127.0.0.1:41482)
0620 20:27:03.934192 (+ 15408us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:03.934215 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:03.934410 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:03.935850 (+ 1440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:03.935861 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:03.935910 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:03.936371 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:03.936382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.938277 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.938282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:03.940903 (+ 2621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:03.940917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:03.943076 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:03.943083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:03.943262 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:03.943292 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 20:27:03.943368 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 20:27:03.943514 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15222,"mutex_wait_us":146,"spinlock_wait_cycles":19456,"thread_start_us":15503,"threads_started":1}
I20260620 20:27:03.947136 18061 tablet_service.cc:1467] Processing CreateTablet for tablet 7f66c8f1da3a4667ad8bb403de9244af (DEFAULT_TABLE table=test-table [id=aeb53eba88f143168109c3cd53754f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:27:03.948503 18061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f66c8f1da3a4667ad8bb403de9244af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:03.948352 18322 tablet_service.cc:1467] Processing CreateTablet for tablet 7f66c8f1da3a4667ad8bb403de9244af (DEFAULT_TABLE table=test-table [id=aeb53eba88f143168109c3cd53754f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:27:03.948350 18192 tablet_service.cc:1467] Processing CreateTablet for tablet 7f66c8f1da3a4667ad8bb403de9244af (DEFAULT_TABLE table=test-table [id=aeb53eba88f143168109c3cd53754f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:27:03.949699 18322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f66c8f1da3a4667ad8bb403de9244af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:03.949702 18192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f66c8f1da3a4667ad8bb403de9244af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:03.953863 18126 heartbeater.cc:503] Master 127.31.235.190:45741 was elected leader, sending a full tablet report...
I20260620 20:27:03.960003 18405 tablet_bootstrap.cc:492] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4: Bootstrap starting.
I20260620 20:27:03.964174 18407 tablet_bootstrap.cc:492] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575: Bootstrap starting.
I20260620 20:27:03.964668 18405 tablet_bootstrap.cc:654] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:03.966086 18405 log.cc:824] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:03.967514 18406 tablet_bootstrap.cc:492] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256: Bootstrap starting.
I20260620 20:27:03.967825 18407 tablet_bootstrap.cc:654] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:03.969218 18407 log.cc:824] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:03.969566 18405 tablet_bootstrap.cc:492] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4: No bootstrap required, opened a new log
I20260620 20:27:03.969864 18405 ts_tablet_manager.cc:1397] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260620 20:27:03.971194 18406 tablet_bootstrap.cc:654] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:03.972474 18406 log.cc:824] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:03.973248 18407 tablet_bootstrap.cc:492] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575: No bootstrap required, opened a new log
I20260620 20:27:03.973556 18407 ts_tablet_manager.cc:1397] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 20:27:03.975703 18406 tablet_bootstrap.cc:492] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256: No bootstrap required, opened a new log
I20260620 20:27:03.975967 18406 ts_tablet_manager.cc:1397] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 20:27:03.978317 18405 raft_consensus.cc:348] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e49a4bf8094a66ad05f5fe4b2a01f4" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 39663 } } peers { permanent_uuid: "3c248de25fbe42308ed8c33b28886256" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41247 } } peers { permanent_uuid: "502b8ea254a747059f81a400d3d57575" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39007 } }
I20260620 20:27:03.978679 18405 raft_consensus.cc:374] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:03.978775 18405 raft_consensus.cc:729] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e49a4bf8094a66ad05f5fe4b2a01f4, State: Initialized, Role: FOLLOWER
I20260620 20:27:03.979359 18405 consensus_queue.cc:260] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e49a4bf8094a66ad05f5fe4b2a01f4" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 39663 } } peers { permanent_uuid: "3c248de25fbe42308ed8c33b28886256" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41247 } } peers { permanent_uuid: "502b8ea254a747059f81a400d3d57575" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39007 } }
I20260620 20:27:03.982149 18405 ts_tablet_manager.cc:1428] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4: Time spent starting tablet: real 0.012s user 0.005s sys 0.006s
I20260620 20:27:03.984615 18407 raft_consensus.cc:348] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e49a4bf8094a66ad05f5fe4b2a01f4" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 39663 } } peers { permanent_uuid: "3c248de25fbe42308ed8c33b28886256" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41247 } } peers { permanent_uuid: "502b8ea254a747059f81a400d3d57575" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39007 } }
I20260620 20:27:03.985081 18407 raft_consensus.cc:374] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:03.985185 18407 raft_consensus.cc:729] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502b8ea254a747059f81a400d3d57575, State: Initialized, Role: FOLLOWER
I20260620 20:27:03.985813 18407 consensus_queue.cc:260] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e49a4bf8094a66ad05f5fe4b2a01f4" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 39663 } } peers { permanent_uuid: "3c248de25fbe42308ed8c33b28886256" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41247 } } peers { permanent_uuid: "502b8ea254a747059f81a400d3d57575" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39007 } }
I20260620 20:27:03.986425 18406 raft_consensus.cc:348] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e49a4bf8094a66ad05f5fe4b2a01f4" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 39663 } } peers { permanent_uuid: "3c248de25fbe42308ed8c33b28886256" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41247 } } peers { permanent_uuid: "502b8ea254a747059f81a400d3d57575" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39007 } }
I20260620 20:27:03.986944 18406 raft_consensus.cc:374] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:03.987066 18406 raft_consensus.cc:729] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c248de25fbe42308ed8c33b28886256, State: Initialized, Role: FOLLOWER
I20260620 20:27:03.987779 18406 consensus_queue.cc:260] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e49a4bf8094a66ad05f5fe4b2a01f4" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 39663 } } peers { permanent_uuid: "3c248de25fbe42308ed8c33b28886256" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41247 } } peers { permanent_uuid: "502b8ea254a747059f81a400d3d57575" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39007 } }
I20260620 20:27:03.988915 18387 heartbeater.cc:503] Master 127.31.235.190:45741 was elected leader, sending a full tablet report...
I20260620 20:27:03.989955 18406 ts_tablet_manager.cc:1428] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260620 20:27:03.990974 18257 heartbeater.cc:503] Master 127.31.235.190:45741 was elected leader, sending a full tablet report...
I20260620 20:27:03.991461 18407 ts_tablet_manager.cc:1428] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575: Time spent starting tablet: real 0.018s user 0.006s sys 0.010s
W20260620 20:27:04.110780 18388 tablet.cc:2369] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:27:04.140352 18258 tablet.cc:2369] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:27:04.147163 18411 raft_consensus.cc:482] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:27:04.147485 18411 raft_consensus.cc:504] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e49a4bf8094a66ad05f5fe4b2a01f4" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 39663 } } peers { permanent_uuid: "3c248de25fbe42308ed8c33b28886256" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41247 } } peers { permanent_uuid: "502b8ea254a747059f81a400d3d57575" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39007 } }
I20260620 20:27:04.149281 18411 leader_election.cc:288] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c248de25fbe42308ed8c33b28886256 (127.31.235.131:41247), 502b8ea254a747059f81a400d3d57575 (127.31.235.130:39007)
I20260620 20:27:04.156141 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:04.149937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:52393 (local address 127.31.235.130:39007)
0620 20:27:04.150176 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:04.150186 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:04.150861 (+ 675us) server_negotiation.cc:407] Connection header received
0620 20:27:04.151087 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:04.151092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:04.151162 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:04.151268 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:04.151873 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:04.152568 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:04.154478 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:04.155246 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:04.155405 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:04.155474 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:04.155740 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 20:27:04.155863 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 20:27:04.156423 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:04.149595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:39007 (local address 127.31.235.129:52393)
0620 20:27:04.150739 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:04.150759 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:04.150937 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:04.151284 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:04.151289 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:04.151312 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:04.151723 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:04.151737 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:04.152694 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:04.152702 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:04.154293 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:04.154305 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:04.155432 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:04.155439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:04.155584 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:04.155608 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:27:04.155722 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 20:27:04.155854 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1014,"mutex_wait_us":39,"thread_start_us":829,"threads_started":1}
I20260620 20:27:04.156536 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:04.149894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:46919 (local address 127.31.235.131:41247)
0620 20:27:04.150168 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:04.150177 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:04.150993 (+ 816us) server_negotiation.cc:407] Connection header received
0620 20:27:04.151214 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:04.151218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:04.151282 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:04.151387 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:04.152092 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:04.152823 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:04.154749 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:04.155567 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:04.155761 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:04.155839 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:04.156101 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 20:27:04.156241 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 20:27:04.156821 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:04.149829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:41247 (local address 127.31.235.129:46919)
0620 20:27:04.150872 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:04.150889 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:04.151056 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:04.151403 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:04.151408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:04.151451 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:04.151916 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:04.151927 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:04.152969 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:04.152976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:04.154553 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:04.154566 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:04.155763 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:04.155770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:04.155937 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:04.155960 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:27:04.156075 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 20:27:04.156210 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"mutex_wait_us":25,"thread_start_us":536,"threads_started":1}
I20260620 20:27:04.157495 18212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f66c8f1da3a4667ad8bb403de9244af" candidate_uuid: "55e49a4bf8094a66ad05f5fe4b2a01f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502b8ea254a747059f81a400d3d57575" is_pre_election: true
I20260620 20:27:04.157970 18342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f66c8f1da3a4667ad8bb403de9244af" candidate_uuid: "55e49a4bf8094a66ad05f5fe4b2a01f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c248de25fbe42308ed8c33b28886256" is_pre_election: true
I20260620 20:27:04.158152 18212 raft_consensus.cc:2454] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55e49a4bf8094a66ad05f5fe4b2a01f4 in term 0.
I20260620 20:27:04.158521 18342 raft_consensus.cc:2454] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55e49a4bf8094a66ad05f5fe4b2a01f4 in term 0.
I20260620 20:27:04.158878 18017 leader_election.cc:302] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 502b8ea254a747059f81a400d3d57575, 55e49a4bf8094a66ad05f5fe4b2a01f4; no voters:
I20260620 20:27:04.159302 18411 raft_consensus.cc:2781] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:27:04.159446 18411 raft_consensus.cc:482] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:27:04.159564 18411 raft_consensus.cc:3037] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:04.161809 18411 raft_consensus.cc:504] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e49a4bf8094a66ad05f5fe4b2a01f4" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 39663 } } peers { permanent_uuid: "3c248de25fbe42308ed8c33b28886256" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41247 } } peers { permanent_uuid: "502b8ea254a747059f81a400d3d57575" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39007 } }
I20260620 20:27:04.162465 18411 leader_election.cc:288] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 [CANDIDATE]: Term 1 election: Requested vote from peers 3c248de25fbe42308ed8c33b28886256 (127.31.235.131:41247), 502b8ea254a747059f81a400d3d57575 (127.31.235.130:39007)
I20260620 20:27:04.162922 18342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f66c8f1da3a4667ad8bb403de9244af" candidate_uuid: "55e49a4bf8094a66ad05f5fe4b2a01f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c248de25fbe42308ed8c33b28886256"
I20260620 20:27:04.162989 18212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f66c8f1da3a4667ad8bb403de9244af" candidate_uuid: "55e49a4bf8094a66ad05f5fe4b2a01f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502b8ea254a747059f81a400d3d57575"
I20260620 20:27:04.163133 18342 raft_consensus.cc:3037] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:04.163196 18212 raft_consensus.cc:3037] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:04.165283 18342 raft_consensus.cc:2454] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55e49a4bf8094a66ad05f5fe4b2a01f4 in term 1.
I20260620 20:27:04.165444 18212 raft_consensus.cc:2454] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55e49a4bf8094a66ad05f5fe4b2a01f4 in term 1.
I20260620 20:27:04.165729 18015 leader_election.cc:302] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c248de25fbe42308ed8c33b28886256, 55e49a4bf8094a66ad05f5fe4b2a01f4; no voters:
I20260620 20:27:04.166039 18411 raft_consensus.cc:2781] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:04.166467 18411 raft_consensus.cc:686] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 [term 1 LEADER]: Becoming Leader. State: Replica: 55e49a4bf8094a66ad05f5fe4b2a01f4, State: Running, Role: LEADER
I20260620 20:27:04.166949 18411 consensus_queue.cc:237] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e49a4bf8094a66ad05f5fe4b2a01f4" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 39663 } } peers { permanent_uuid: "3c248de25fbe42308ed8c33b28886256" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41247 } } peers { permanent_uuid: "502b8ea254a747059f81a400d3d57575" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39007 } }
I20260620 20:27:04.171483 17939 catalog_manager.cc:5495] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55e49a4bf8094a66ad05f5fe4b2a01f4 (127.31.235.129). New cstate: current_term: 1 leader_uuid: "55e49a4bf8094a66ad05f5fe4b2a01f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e49a4bf8094a66ad05f5fe4b2a01f4" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 39663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c248de25fbe42308ed8c33b28886256" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "502b8ea254a747059f81a400d3d57575" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39007 } health_report { overall_health: UNKNOWN } } }
W20260620 20:27:04.178447 18127 tablet.cc:2369] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:27:04.240658 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:04.234380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.31.235.129:39663)
0620 20:27:04.234604 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:04.234611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:04.234690 (+ 79us) server_negotiation.cc:407] Connection header received
0620 20:27:04.234993 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:04.234998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:04.235110 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:04.235305 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:27:04.235673 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:04.236502 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:04.238450 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:04.238886 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:04.239019 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:04.239553 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:27:04.240062 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:27:04.240240 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:04.240308 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 20:27:04.240391 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:27:04.249308 18342 raft_consensus.cc:1261] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256 [term 1 FOLLOWER]: Refusing update from remote peer 55e49a4bf8094a66ad05f5fe4b2a01f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:27:04.249297 18212 raft_consensus.cc:1261] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575 [term 1 FOLLOWER]: Refusing update from remote peer 55e49a4bf8094a66ad05f5fe4b2a01f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:27:04.250385 18416 consensus_queue.cc:1035] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c248de25fbe42308ed8c33b28886256" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 41247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:27:04.250779 18411 consensus_queue.cc:1035] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "502b8ea254a747059f81a400d3d57575" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 39007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:27:04.268239 18421 mvcc.cc:205] Tried to move back new op lower bound from 7299019670514180096 to 7299019670191280128. Current Snapshot: MvccSnapshot[applied={T|T < 7299019670514180096}]
I20260620 20:27:04.285229 17940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:27:04.285692 17940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:27:04.289302 17940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ee8828ad1444664bc4cb61804e46112: Sending DeleteTablet for 3 replicas of tablet 7f66c8f1da3a4667ad8bb403de9244af
I20260620 20:27:04.290669 18061 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f66c8f1da3a4667ad8bb403de9244af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:27:04 UTC) from {username='slave'} at 127.0.0.1:37000
I20260620 20:27:04.290799 18322 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f66c8f1da3a4667ad8bb403de9244af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:27:04 UTC) from {username='slave'} at 127.0.0.1:37346
I20260620 20:27:04.292068 18192 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f66c8f1da3a4667ad8bb403de9244af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:27:04 UTC) from {username='slave'} at 127.0.0.1:41482
I20260620 20:27:04.293136 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 17999
I20260620 20:27:04.301267 18429 tablet_replica.cc:330] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4: stopping tablet replica
I20260620 20:27:04.301859 18429 raft_consensus.cc:2229] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:27:04.303284 18429 raft_consensus.cc:2258] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:27:04.306437 18429 ts_tablet_manager.cc:1905] T 7f66c8f1da3a4667ad8bb403de9244af P 55e49a4bf8094a66ad05f5fe4b2a01f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:27:04.309620 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:04.294458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37010 (local address 127.31.235.129:39663)
0620 20:27:04.294702 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:04.294710 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:04.300833 (+ 6123us) server_negotiation.cc:407] Connection header received
0620 20:27:04.301208 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:04.301214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:04.301333 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:04.301420 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:04.302128 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:04.303407 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:04.305363 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:04.306050 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:04.306151 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:04.307717 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:04.307728 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:04.307746 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:04.308603 (+ 857us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:04.308670 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:04.308704 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:04.308784 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:04.309037 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 20:27:04.309162 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:27:04.310757 18121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:04.314126 18431 tablet_replica.cc:330] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575: stopping tablet replica
I20260620 20:27:04.314708 18431 raft_consensus.cc:2229] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:27:04.315400 18431 raft_consensus.cc:2258] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:27:04.316218 18430 tablet_replica.cc:330] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256: stopping tablet replica
I20260620 20:27:04.316519 18430 raft_consensus.cc:2229] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:27:04.316859 18430 raft_consensus.cc:2258] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:27:04.318579 18431 ts_tablet_manager.cc:1905] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:27:04.319259 18430 ts_tablet_manager.cc:1905] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:27:04.325531 18431 ts_tablet_manager.cc:1918] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:27:04.325690 18431 log.cc:1196] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/7f66c8f1da3a4667ad8bb403de9244af
I20260620 20:27:04.326184 18431 ts_tablet_manager.cc:1939] T 7f66c8f1da3a4667ad8bb403de9244af P 502b8ea254a747059f81a400d3d57575: Deleting consensus metadata
I20260620 20:27:04.326817 18430 ts_tablet_manager.cc:1918] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:27:04.326982 18430 log.cc:1196] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/7f66c8f1da3a4667ad8bb403de9244af
I20260620 20:27:04.327561 18430 ts_tablet_manager.cc:1939] T 7f66c8f1da3a4667ad8bb403de9244af P 3c248de25fbe42308ed8c33b28886256: Deleting consensus metadata
I20260620 20:27:04.328045 17927 catalog_manager.cc:4841] TS 502b8ea254a747059f81a400d3d57575 (127.31.235.130:39007): tablet 7f66c8f1da3a4667ad8bb403de9244af (table test-table [id=aeb53eba88f143168109c3cd53754f4a]) successfully deleted
I20260620 20:27:04.329128 17925 catalog_manager.cc:4841] TS 3c248de25fbe42308ed8c33b28886256 (127.31.235.131:41247): tablet 7f66c8f1da3a4667ad8bb403de9244af (table test-table [id=aeb53eba88f143168109c3cd53754f4a]) successfully deleted
I20260620 20:27:04.430457 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 17999
I20260620 20:27:04.449064 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 18130
W20260620 20:27:04.449272 17925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.235.129:39663 (error 108)
I20260620 20:27:04.451212 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:04.450528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:39663 (local address 127.0.0.1:37012)
0620 20:27:04.450943 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:04.451060 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:39663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":214,"threads_started":1}
W20260620 20:27:04.451462 17925 catalog_manager.cc:4584] TS 55e49a4bf8094a66ad05f5fe4b2a01f4 (127.31.235.129:39663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f66c8f1da3a4667ad8bb403de9244af: Network error: Client connection negotiation failed: client connection to 127.31.235.129:39663: connect: Connection refused (error 111)
I20260620 20:27:04.456621 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:04.450184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.31.235.130:39007)
0620 20:27:04.450383 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:04.450389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:04.450412 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:27:04.450465 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:04.450469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:04.450542 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:04.450651 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:04.451022 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:04.451905 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:04.453369 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:04.454021 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:04.454157 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:04.455232 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:04.455243 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:04.455262 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:04.455868 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:04.455914 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:04.455933 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:04.456051 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:04.456178 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 20:27:04.456284 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:27:04.457278 18252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:04.505386 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:04.504781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:39663 (local address 127.0.0.1:37022)
0620 20:27:04.505067 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:04.505189 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:39663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 20:27:04.506686 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:04.506217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:39663 (local address 127.0.0.1:37038)
0620 20:27:04.506387 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:04.506481 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:39663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 20:27:04.548446 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 18130
I20260620 20:27:04.565143 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 18260
I20260620 20:27:04.567269 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:04.566804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:39663 (local address 127.0.0.1:37050)
0620 20:27:04.566976 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:04.567080 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:39663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 20:27:04.568364 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:04.567930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:39663 (local address 127.0.0.1:37054)
0620 20:27:04.568090 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:04.568187 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:39663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 20:27:04.572736 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:04.566038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.31.235.131:41247)
0620 20:27:04.566213 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:04.566217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:04.566238 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:27:04.566425 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:04.566429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:04.566514 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:04.566602 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:04.567130 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:04.568183 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:04.569808 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:04.570203 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:04.570358 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:04.571423 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:04.571432 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:04.571447 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:04.572039 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:04.572090 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:04.572110 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:04.572198 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:04.572382 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 20:27:04.572481 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:27:04.573406 18382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:04.648942 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:04.648462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:39663 (local address 127.0.0.1:37058)
0620 20:27:04.648691 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:04.648782 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:39663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 20:27:04.650061 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:04.649644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:39663 (local address 127.0.0.1:37072)
0620 20:27:04.649856 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:04.649922 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:39663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 20:27:04.663787 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 18260
I20260620 20:27:04.680532 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 17909
I20260620 20:27:04.681808 17970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:04.794312 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 17909
2026-06-20T20:27:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2960 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:27:04 dist-test-slave-zzwf krb5kdc[18441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:27:04 dist-test-slave-zzwf krb5kdc[18441](info): set up 2 sockets
Jun 20 20:27:04 dist-test-slave-zzwf krb5kdc[18441](info): commencing operation
krb5kdc: starting...
W20260620 20:27:05.974120 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:27:06 dist-test-slave-zzwf krb5kdc[18441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:27:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:27:06.224517 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:35041
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:35041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:06.556527 18454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:06.556882 18454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:06.557055 18454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:06.564327 18454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:06.564482 18454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:06.564595 18454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:06.564702 18454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:06.564790 18454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:06.574628 18454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:35041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:35041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:06.576082 18454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:06.577376 18454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:06.589399 18459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:06.589433 18462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:06.589372 18460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:06.590220 18454 server_base.cc:1034] running on GCE node
I20260620 20:27:06.590993 18454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:06.592391 18454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:06.593597 18454 hybrid_clock.cc:648] HybridClock initialized: now 1781987226593540 us; error 42 us; skew 500 ppm
I20260620 20:27:06.593974 18454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:06 dist-test-slave-zzwf krb5kdc[18441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987226, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:06.597107 18454 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:27:06.598809 18454 webserver.cc:460] Webserver started at http://127.31.235.190:36065/ using document root <none> and password file <none>
I20260620 20:27:06.599555 18454 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:06.599707 18454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:06.600030 18454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:06.603546 18454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "4c994e05dc5e4eb3a44bc460e5a587c8"
format_stamp: "Formatted at 2026-06-20 20:27:06 on dist-test-slave-zzwf"
server_key: "9597d65972d5ad2f5486d715b68f19b6"
server_key_iv: "4a1994ff98bf33eba5304e5deb42a911"
server_key_version: "encryptionkey@0"
I20260620 20:27:06.604311 18454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "4c994e05dc5e4eb3a44bc460e5a587c8"
format_stamp: "Formatted at 2026-06-20 20:27:06 on dist-test-slave-zzwf"
server_key: "9597d65972d5ad2f5486d715b68f19b6"
server_key_iv: "4a1994ff98bf33eba5304e5deb42a911"
server_key_version: "encryptionkey@0"
I20260620 20:27:06.609678 18454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 20:27:06.613277 18469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:06.614691 18454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:27:06.614899 18454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "4c994e05dc5e4eb3a44bc460e5a587c8"
format_stamp: "Formatted at 2026-06-20 20:27:06 on dist-test-slave-zzwf"
server_key: "9597d65972d5ad2f5486d715b68f19b6"
server_key_iv: "4a1994ff98bf33eba5304e5deb42a911"
server_key_version: "encryptionkey@0"
I20260620 20:27:06.615134 18454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:06.648203 18454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:06.652518 18454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:06.652832 18454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:06.671941 18454 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:35041
I20260620 20:27:06.671972 18520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:35041 every 8 connection(s)
I20260620 20:27:06.673369 18454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:06.677402 18521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:06.679519 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 18454
I20260620 20:27:06.679731 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:06.680191 32686 external_mini_cluster.cc:1428] Setting key bfbdfc7358ff87057eacfd3f9ca5339c
I20260620 20:27:06.689934 18521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8: Bootstrap starting.
I20260620 20:27:06.694006 18521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:06.695248 18521 log.cc:824] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:27:06 dist-test-slave-zzwf krb5kdc[18441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:27:06.698869 18521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8: No bootstrap required, opened a new log
I20260620 20:27:06.705374 18524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:06.682111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58540 (local address 127.31.235.190:35041)
0620 20:27:06.682936 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:06.682948 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:06.682999 (+ 51us) server_negotiation.cc:407] Connection header received
0620 20:27:06.684673 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:06.684696 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:06.685096 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:06.685415 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:06.687265 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:06.688624 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:06.690499 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:06.691171 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:06.691342 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:06.696705 (+ 5363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:06.696718 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:06.696737 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:06.696795 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:06.700789 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:06.701551 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:06.701560 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:06.701570 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:06.701804 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:06.702233 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:06.702238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:06.702242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:06.702933 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:06.703263 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:06.703581 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 20:27:06.704006 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":378,"threads_started":1}
I20260620 20:27:06.708817 18521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c994e05dc5e4eb3a44bc460e5a587c8" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35041 } }
I20260620 20:27:06.709126 18521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:06.709196 18521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c994e05dc5e4eb3a44bc460e5a587c8, State: Initialized, Role: FOLLOWER
I20260620 20:27:06.709728 18521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c994e05dc5e4eb3a44bc460e5a587c8" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35041 } }
I20260620 20:27:06.709954 18521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:06.710057 18521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:06.710173 18521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:06.712265 18521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c994e05dc5e4eb3a44bc460e5a587c8" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35041 } }
I20260620 20:27:06.712675 18521 leader_election.cc:302] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c994e05dc5e4eb3a44bc460e5a587c8; no voters:
I20260620 20:27:06.713183 18521 leader_election.cc:288] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:06.713361 18526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:06.714437 18526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8 [term 1 LEADER]: Becoming Leader. State: Replica: 4c994e05dc5e4eb3a44bc460e5a587c8, State: Running, Role: LEADER
I20260620 20:27:06.715008 18526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c994e05dc5e4eb3a44bc460e5a587c8" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35041 } }
I20260620 20:27:06.715442 18521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:06.718070 18527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c994e05dc5e4eb3a44bc460e5a587c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c994e05dc5e4eb3a44bc460e5a587c8" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35041 } } }
I20260620 20:27:06.718559 18527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:06.719089 18528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c994e05dc5e4eb3a44bc460e5a587c8. Latest consensus state: current_term: 1 leader_uuid: "4c994e05dc5e4eb3a44bc460e5a587c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c994e05dc5e4eb3a44bc460e5a587c8" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 35041 } } }
I20260620 20:27:06.719432 18528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:06.720558 18533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:06.726039 18533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:06.734310 18533 catalog_manager.cc:1347] Generated new cluster ID: 74aea90ccec14756baedae5975794d0b
I20260620 20:27:06.734423 18533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:06.757138 18533 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:06.759155 18533 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:06.781350 18533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8: Generated new TSK 0
I20260620 20:27:06.782536 18533 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
I20260620 20:27:06.841216 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:35041
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:07.179244 18547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:07.179539 18547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:07.179695 18547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:07.187345 18547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:07.187456 18547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:07.187624 18547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:27:07.197606 18547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:35041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:07.198889 18547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:07.200183 18547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:07.214223 18553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:07.214249 18552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:07.214277 18555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:07.215166 18547 server_base.cc:1034] running on GCE node
I20260620 20:27:07.215827 18547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:07.216868 18547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:07.218088 18547 hybrid_clock.cc:648] HybridClock initialized: now 1781987227218045 us; error 69 us; skew 500 ppm
I20260620 20:27:07.218439 18547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:07 dist-test-slave-zzwf krb5kdc[18441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987227, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:07.221961 18547 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:27:07.223570 18547 webserver.cc:460] Webserver started at http://127.31.235.129:42709/ using document root <none> and password file <none>
I20260620 20:27:07.224249 18547 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:07.224396 18547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:07.224710 18547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:07.227641 18547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "4c03af821d3e49b5b80e3936c3b8b484"
format_stamp: "Formatted at 2026-06-20 20:27:07 on dist-test-slave-zzwf"
server_key: "a02effe49b325f260d60372cfced3719"
server_key_iv: "9b3535b424ee638b96af7b3673986e20"
server_key_version: "encryptionkey@0"
I20260620 20:27:07.228379 18547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "4c03af821d3e49b5b80e3936c3b8b484"
format_stamp: "Formatted at 2026-06-20 20:27:07 on dist-test-slave-zzwf"
server_key: "a02effe49b325f260d60372cfced3719"
server_key_iv: "9b3535b424ee638b96af7b3673986e20"
server_key_version: "encryptionkey@0"
I20260620 20:27:07.233415 18547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 20:27:07.236793 18562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:07.238373 18547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:27:07.238543 18547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "4c03af821d3e49b5b80e3936c3b8b484"
format_stamp: "Formatted at 2026-06-20 20:27:07 on dist-test-slave-zzwf"
server_key: "a02effe49b325f260d60372cfced3719"
server_key_iv: "9b3535b424ee638b96af7b3673986e20"
server_key_version: "encryptionkey@0"
I20260620 20:27:07.238739 18547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:07.262428 18547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:07.266767 18547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:07.267071 18547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:07.268060 18547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:27:07.269361 18547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:27:07.269465 18547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:07.269620 18547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:27:07.269692 18547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:07.303117 18547 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:41597
I20260620 20:27:07.303202 18674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:41597 every 8 connection(s)
I20260620 20:27:07.304431 18547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:27:07.309120 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 18547
I20260620 20:27:07.309314 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:27:07.309847 32686 external_mini_cluster.cc:1428] Setting key 8a04d5ceb118750c274a1d06d6c71d33
Jun 20 20:27:07 dist-test-slave-zzwf krb5kdc[18441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987227, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:27:07.325014 18677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:07.306991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:34373 (local address 127.31.235.190:35041)
0620 20:27:07.307455 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:07.307459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:07.308399 (+ 940us) server_negotiation.cc:407] Connection header received
0620 20:27:07.310543 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:07.310550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:07.310650 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:07.310773 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:07.312338 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:07.313076 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:07.314683 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:07.315161 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:07.315296 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:07.318898 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:07.318903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:07.318907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:07.318939 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:07.321666 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:07.322612 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:07.322616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:07.322618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:07.322731 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:07.323227 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:07.323230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:07.323232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:07.323515 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:07.323622 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:07.324553 (+ 931us) server_negotiation.cc:299] Negotiation successful
0620 20:27:07.324734 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":222,"threads_started":1}
I20260620 20:27:07.326537 18678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:07.307543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35041 (local address 127.31.235.129:34373)
0620 20:27:07.308225 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:07.308271 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:07.310234 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:07.311029 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:07.311038 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:07.311564 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:07.312152 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:07.312170 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:07.313247 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:07.313251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:07.314537 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:07.314549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:07.315390 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:07.315398 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:07.315682 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:07.316363 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:27:07.316385 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:07.318667 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:07.321792 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:07.321803 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:07.321831 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:07.322499 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:07.322847 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:07.322855 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:07.322861 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:07.323106 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:07.323641 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:07.323652 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:07.324122 (+ 470us) client_negotiation.cc:769] Sending connection context
0620 20:27:07.324438 (+ 316us) client_negotiation.cc:240] Negotiation successful
0620 20:27:07.325030 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":268,"threads_started":1}
I20260620 20:27:07.328146 18675 heartbeater.cc:347] Connected to a master server at 127.31.235.190:35041
I20260620 20:27:07.328478 18675 heartbeater.cc:464] Registering TS with master...
I20260620 20:27:07.329409 18675 heartbeater.cc:511] Master 127.31.235.190:35041 requested a full tablet report, sending...
I20260620 20:27:07.331768 18486 ts_manager.cc:194] Registered new tserver with Master: 4c03af821d3e49b5b80e3936c3b8b484 (127.31.235.129:41597)
I20260620 20:27:07.333201 18486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:34373
WARNING: no policy specified for kudu/127.31.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
I20260620 20:27:07.367274 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:35041
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:07.715875 18681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:07.716187 18681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:07.716352 18681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:07.724014 18681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:07.724133 18681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:07.724251 18681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:27:07.734197 18681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:35041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:07.735455 18681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:07.736874 18681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:07.751075 18689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:07.751010 18687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:07.751003 18686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:07.751745 18681 server_base.cc:1034] running on GCE node
I20260620 20:27:07.752332 18681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:07.753170 18681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:07.754398 18681 hybrid_clock.cc:648] HybridClock initialized: now 1781987227754350 us; error 43 us; skew 500 ppm
I20260620 20:27:07.754664 18681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:07 dist-test-slave-zzwf krb5kdc[18441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987227, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:07.757802 18681 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:27:07.759233 18681 webserver.cc:460] Webserver started at http://127.31.235.130:36687/ using document root <none> and password file <none>
I20260620 20:27:07.759908 18681 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:07.760031 18681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:07.760362 18681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:07.763237 18681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "e20fd9154e1747669f517301541e189b"
format_stamp: "Formatted at 2026-06-20 20:27:07 on dist-test-slave-zzwf"
server_key: "ab9a1d7399845ef2596191752ef14ede"
server_key_iv: "9250d57996a679ada9a2805503aa8160"
server_key_version: "encryptionkey@0"
I20260620 20:27:07.763971 18681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "e20fd9154e1747669f517301541e189b"
format_stamp: "Formatted at 2026-06-20 20:27:07 on dist-test-slave-zzwf"
server_key: "ab9a1d7399845ef2596191752ef14ede"
server_key_iv: "9250d57996a679ada9a2805503aa8160"
server_key_version: "encryptionkey@0"
I20260620 20:27:07.768901 18681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:27:07.772250 18696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:07.773471 18681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 20:27:07.773701 18681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "e20fd9154e1747669f517301541e189b"
format_stamp: "Formatted at 2026-06-20 20:27:07 on dist-test-slave-zzwf"
server_key: "ab9a1d7399845ef2596191752ef14ede"
server_key_iv: "9250d57996a679ada9a2805503aa8160"
server_key_version: "encryptionkey@0"
I20260620 20:27:07.773952 18681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:07.792811 18681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:07.797113 18681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:07.797428 18681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:07.798449 18681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:27:07.799746 18681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:27:07.799847 18681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:07.799994 18681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:27:07.800065 18681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:07.834751 18681 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:38991
I20260620 20:27:07.834805 18808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:38991 every 8 connection(s)
I20260620 20:27:07.836078 18681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:27:07.842175 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 18681
I20260620 20:27:07.842381 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:27:07.842983 32686 external_mini_cluster.cc:1428] Setting key 81b03759b3ae74d8734bbb5f04db64f4
Jun 20 20:27:07 dist-test-slave-zzwf krb5kdc[18441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987227, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:27:07.861220 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:07.838742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:54051 (local address 127.31.235.190:35041)
0620 20:27:07.839230 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:07.839248 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:07.840212 (+ 964us) server_negotiation.cc:407] Connection header received
0620 20:27:07.842165 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:07.842172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:07.842293 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:07.842429 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:07.844172 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:07.845160 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:07.846808 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:07.847564 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:07.847713 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:07.851498 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:07.851507 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:07.851515 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:07.851564 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:07.857460 (+ 5896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:07.858616 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:07.858622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:07.858624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:07.858779 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:07.859286 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:07.859290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:07.859292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:07.859586 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:07.859722 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:07.860707 (+ 985us) server_negotiation.cc:299] Negotiation successful
0620 20:27:07.860907 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":217,"threads_started":1}
I20260620 20:27:07.862672 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:07.839379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35041 (local address 127.31.235.130:54051)
0620 20:27:07.840052 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:07.840094 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:07.841849 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:07.842669 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:07.842677 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:07.843274 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:07.843996 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:07.844023 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:07.845321 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:07.845327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:07.846646 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:07.846659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:07.847809 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:07.847814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:07.848010 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:07.848708 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:27:07.848734 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:07.851247 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:07.857719 (+ 6472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:07.857748 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:07.857771 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:07.858459 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:07.858899 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:07.858907 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:07.858914 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:07.859157 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:07.859725 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:07.859737 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:07.860232 (+ 495us) client_negotiation.cc:769] Sending connection context
0620 20:27:07.860559 (+ 327us) client_negotiation.cc:240] Negotiation successful
0620 20:27:07.861176 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":261,"threads_started":1}
I20260620 20:27:07.864408 18809 heartbeater.cc:347] Connected to a master server at 127.31.235.190:35041
I20260620 20:27:07.864800 18809 heartbeater.cc:464] Registering TS with master...
I20260620 20:27:07.865782 18809 heartbeater.cc:511] Master 127.31.235.190:35041 requested a full tablet report, sending...
I20260620 20:27:07.867535 18486 ts_manager.cc:194] Registered new tserver with Master: e20fd9154e1747669f517301541e189b (127.31.235.130:38991)
I20260620 20:27:07.868806 18486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:54051
WARNING: no policy specified for kudu/127.31.235.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
I20260620 20:27:07.902004 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:35041
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:08.231930 18815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:08.232219 18815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:08.232379 18815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:08.239753 18815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:08.239857 18815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:08.240065 18815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:27:08.252699 18815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:35041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:08.254565 18815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:08.256172 18815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:08.269402 18820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:08.269518 18821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:08.269523 18823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:08.270538 18815 server_base.cc:1034] running on GCE node
I20260620 20:27:08.271153 18815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:08.272208 18815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:08.273448 18815 hybrid_clock.cc:648] HybridClock initialized: now 1781987228273407 us; error 35 us; skew 500 ppm
I20260620 20:27:08.273798 18815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:08 dist-test-slave-zzwf krb5kdc[18441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987228, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:08.276944 18815 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:27:08.278689 18815 webserver.cc:460] Webserver started at http://127.31.235.131:35115/ using document root <none> and password file <none>
I20260620 20:27:08.279377 18815 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:08.279520 18815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:08.279838 18815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:08.282814 18815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "7520fa4f7fd146bf8aaf8a059ad6cb14"
format_stamp: "Formatted at 2026-06-20 20:27:08 on dist-test-slave-zzwf"
server_key: "46215fb28b8da9ff20ad7ec21a3d223a"
server_key_iv: "82f6ab34c1758cac98d581546bc522bd"
server_key_version: "encryptionkey@0"
I20260620 20:27:08.283573 18815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "7520fa4f7fd146bf8aaf8a059ad6cb14"
format_stamp: "Formatted at 2026-06-20 20:27:08 on dist-test-slave-zzwf"
server_key: "46215fb28b8da9ff20ad7ec21a3d223a"
server_key_iv: "82f6ab34c1758cac98d581546bc522bd"
server_key_version: "encryptionkey@0"
I20260620 20:27:08.288882 18815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 20:27:08.292513 18830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:08.293948 18815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:27:08.294119 18815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "7520fa4f7fd146bf8aaf8a059ad6cb14"
format_stamp: "Formatted at 2026-06-20 20:27:08 on dist-test-slave-zzwf"
server_key: "46215fb28b8da9ff20ad7ec21a3d223a"
server_key_iv: "82f6ab34c1758cac98d581546bc522bd"
server_key_version: "encryptionkey@0"
I20260620 20:27:08.294337 18815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:08.305966 18815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:08.310235 18815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:08.310545 18815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:08.311509 18815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:27:08.312898 18815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:27:08.313007 18815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:08.313161 18815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:27:08.313241 18815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:08.336774 18675 heartbeater.cc:503] Master 127.31.235.190:35041 was elected leader, sending a full tablet report...
I20260620 20:27:08.348570 18815 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:36433
I20260620 20:27:08.348621 18942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:36433 every 8 connection(s)
I20260620 20:27:08.349992 18815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:27:08.356805 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 18815
I20260620 20:27:08.356994 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:27:08.357434 32686 external_mini_cluster.cc:1428] Setting key 6c0b7598a1a783d50a8754e830170810
Jun 20 20:27:08 dist-test-slave-zzwf krb5kdc[18441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987228, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:27:08.370110 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:08.352647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:44485 (local address 127.31.235.190:35041)
0620 20:27:08.352830 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:08.352835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:08.353961 (+ 1126us) server_negotiation.cc:407] Connection header received
0620 20:27:08.355853 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:08.355859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:08.355966 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:08.356085 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:08.357781 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.358763 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.360361 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.360800 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.360973 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:08.364357 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:08.364363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:08.364366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:08.364400 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:08.367017 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:08.367764 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:08.367769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:08.367771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:08.367888 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:08.368336 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:08.368345 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:08.368347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:08.368620 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:08.368795 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:08.369534 (+ 739us) server_negotiation.cc:299] Negotiation successful
0620 20:27:08.369769 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:27:08.371068 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:08.353053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:35041 (local address 127.31.235.131:44485)
0620 20:27:08.353796 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:08.353839 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:08.355568 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:08.356323 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:08.356332 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:08.356849 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:08.357607 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:08.357627 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.358912 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.358918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:08.360198 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:08.360210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.361028 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.361037 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:08.361281 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:08.361975 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:27:08.361998 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:08.364140 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:08.367161 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:08.367170 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:08.367189 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:08.367628 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:08.368008 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:08.368013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:08.368017 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:08.368210 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:08.368763 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:08.368773 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:08.369120 (+ 347us) client_negotiation.cc:769] Sending connection context
0620 20:27:08.369391 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 20:27:08.369969 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":277,"threads_started":1}
I20260620 20:27:08.372586 18943 heartbeater.cc:347] Connected to a master server at 127.31.235.190:35041
I20260620 20:27:08.372939 18943 heartbeater.cc:464] Registering TS with master...
I20260620 20:27:08.373692 18943 heartbeater.cc:511] Master 127.31.235.190:35041 requested a full tablet report, sending...
I20260620 20:27:08.375079 18486 ts_manager.cc:194] Registered new tserver with Master: 7520fa4f7fd146bf8aaf8a059ad6cb14 (127.31.235.131:36433)
I20260620 20:27:08.375880 18486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:44485
I20260620 20:27:08.385218 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:27:08.402405 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:08.389261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58544 (local address 127.31.235.190:35041)
0620 20:27:08.389459 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:08.389464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:08.389932 (+ 468us) server_negotiation.cc:407] Connection header received
0620 20:27:08.390082 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:08.390086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:08.390152 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:08.390248 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:08.391663 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.392418 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.393788 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.394251 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.394356 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:08.396990 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:08.396995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:08.396998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:08.397031 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:08.399923 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:08.400631 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:08.400635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:08.400638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:08.400757 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:08.401160 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:08.401164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:08.401166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:08.401448 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:08.401691 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:08.402027 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 20:27:08.402155 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:27:08.406591 18486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:27:08.413475 18486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:27:08.448463 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:08.425954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.31.235.129:41597)
0620 20:27:08.438437 (+ 12483us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:08.438451 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:08.439310 (+ 859us) server_negotiation.cc:407] Connection header received
0620 20:27:08.440263 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:08.440271 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:08.440727 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:08.440924 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:08.442058 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.443368 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.445867 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.446832 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.447087 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:08.447273 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:08.447914 (+ 641us) server_negotiation.cc:299] Negotiation successful
0620 20:27:08.448096 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12207,"thread_start_us":12514,"threads_started":1}
I20260620 20:27:08.449553 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:08.426532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32846 (local address 127.31.235.130:38991)
0620 20:27:08.440089 (+ 13557us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:08.440098 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:08.440128 (+ 30us) server_negotiation.cc:407] Connection header received
0620 20:27:08.440231 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:08.440239 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:08.440522 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:08.440731 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:08.442097 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.443721 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.446974 (+ 3253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.448298 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.448498 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:08.448660 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:08.449064 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 20:27:08.449215 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13388,"thread_start_us":13932,"threads_started":1}
I20260620 20:27:08.449569 18957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:08.425794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:41597 (local address 127.0.0.1:41384)
0620 20:27:08.439099 (+ 13305us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:08.439149 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:08.440054 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:08.441168 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:08.441173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:08.441215 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:08.441886 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:08.441900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.443536 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.443542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:08.445717 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:08.445729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.447437 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.447444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:08.447643 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:08.447698 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 20:27:08.448842 (+ 1144us) client_negotiation.cc:240] Negotiation successful
0620 20:27:08.449055 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12986,"mutex_wait_us":3,"spinlock_wait_cycles":14208,"thread_start_us":13093,"threads_started":1}
I20260620 20:27:08.449633 18955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:08.426020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:38991 (local address 127.0.0.1:32846)
0620 20:27:08.439184 (+ 13164us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:08.439208 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:08.439442 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:08.441028 (+ 1586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:08.441036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:08.441085 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:08.441834 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:08.441857 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.443911 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.443916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:08.446769 (+ 2853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:08.446784 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.448525 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.448531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:08.448873 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:08.448897 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:27:08.449010 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 20:27:08.449137 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13006,"mutex_wait_us":157,"spinlock_wait_cycles":7808,"thread_start_us":12627,"threads_started":1}
I20260620 20:27:08.450428 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:08.427004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36362 (local address 127.31.235.131:36433)
0620 20:27:08.439641 (+ 12637us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:08.439651 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:08.440328 (+ 677us) server_negotiation.cc:407] Connection header received
0620 20:27:08.440427 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:08.440434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:08.440619 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:08.440771 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:08.442117 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.443778 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.446955 (+ 3177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.448359 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.448555 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:08.448729 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:08.449884 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0620 20:27:08.450112 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12435,"thread_start_us":12773,"threads_started":1}
I20260620 20:27:08.449781 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:08.426343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:36433 (local address 127.0.0.1:36362)
0620 20:27:08.440082 (+ 13739us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:08.440099 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:08.440219 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:08.441028 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:08.441036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:08.441079 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:08.441933 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:08.441942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.443917 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.443921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:08.446777 (+ 2856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:08.446785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.448883 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.448889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:08.449084 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:08.449114 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 20:27:08.449200 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 20:27:08.449326 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13620,"mutex_wait_us":180,"spinlock_wait_cycles":12032,"thread_start_us":13453,"threads_started":1}
I20260620 20:27:08.453105 18744 tablet_service.cc:1467] Processing CreateTablet for tablet 0a012a6429e949109ba4806330aca1bf (DEFAULT_TABLE table=test-table [id=4f53cae3662e418eabed843523f2f4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:27:08.453107 18610 tablet_service.cc:1467] Processing CreateTablet for tablet 0a012a6429e949109ba4806330aca1bf (DEFAULT_TABLE table=test-table [id=4f53cae3662e418eabed843523f2f4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:27:08.454589 18610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a012a6429e949109ba4806330aca1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:08.454931 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a012a6429e949109ba4806330aca1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:08.457108 18878 tablet_service.cc:1467] Processing CreateTablet for tablet 0a012a6429e949109ba4806330aca1bf (DEFAULT_TABLE table=test-table [id=4f53cae3662e418eabed843523f2f4e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:27:08.458624 18878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a012a6429e949109ba4806330aca1bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:08.467255 18961 tablet_bootstrap.cc:492] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b: Bootstrap starting.
I20260620 20:27:08.470175 18962 tablet_bootstrap.cc:492] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484: Bootstrap starting.
I20260620 20:27:08.470734 18961 tablet_bootstrap.cc:654] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:08.472057 18961 log.cc:824] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:08.474133 18964 tablet_bootstrap.cc:492] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14: Bootstrap starting.
I20260620 20:27:08.474753 18962 tablet_bootstrap.cc:654] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:08.475328 18961 tablet_bootstrap.cc:492] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b: No bootstrap required, opened a new log
I20260620 20:27:08.475600 18961 ts_tablet_manager.cc:1397] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 20:27:08.476009 18962 log.cc:824] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:08.477056 18964 tablet_bootstrap.cc:654] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:08.478241 18964 log.cc:824] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:08.479185 18962 tablet_bootstrap.cc:492] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484: No bootstrap required, opened a new log
I20260620 20:27:08.479460 18962 ts_tablet_manager.cc:1397] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260620 20:27:08.480718 18964 tablet_bootstrap.cc:492] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14: No bootstrap required, opened a new log
I20260620 20:27:08.481003 18964 ts_tablet_manager.cc:1397] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260620 20:27:08.486192 18961 raft_consensus.cc:348] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c03af821d3e49b5b80e3936c3b8b484" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41597 } } peers { permanent_uuid: "e20fd9154e1747669f517301541e189b" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38991 } } peers { permanent_uuid: "7520fa4f7fd146bf8aaf8a059ad6cb14" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36433 } }
I20260620 20:27:08.486672 18961 raft_consensus.cc:374] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:08.486770 18961 raft_consensus.cc:729] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e20fd9154e1747669f517301541e189b, State: Initialized, Role: FOLLOWER
I20260620 20:27:08.487411 18961 consensus_queue.cc:260] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c03af821d3e49b5b80e3936c3b8b484" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41597 } } peers { permanent_uuid: "e20fd9154e1747669f517301541e189b" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38991 } } peers { permanent_uuid: "7520fa4f7fd146bf8aaf8a059ad6cb14" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36433 } }
I20260620 20:27:08.488595 18809 heartbeater.cc:503] Master 127.31.235.190:35041 was elected leader, sending a full tablet report...
I20260620 20:27:08.489233 18961 ts_tablet_manager.cc:1428] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260620 20:27:08.489456 18964 raft_consensus.cc:348] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c03af821d3e49b5b80e3936c3b8b484" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41597 } } peers { permanent_uuid: "e20fd9154e1747669f517301541e189b" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38991 } } peers { permanent_uuid: "7520fa4f7fd146bf8aaf8a059ad6cb14" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36433 } }
I20260620 20:27:08.489895 18964 raft_consensus.cc:374] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:08.489979 18964 raft_consensus.cc:729] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7520fa4f7fd146bf8aaf8a059ad6cb14, State: Initialized, Role: FOLLOWER
I20260620 20:27:08.489948 18962 raft_consensus.cc:348] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c03af821d3e49b5b80e3936c3b8b484" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41597 } } peers { permanent_uuid: "e20fd9154e1747669f517301541e189b" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38991 } } peers { permanent_uuid: "7520fa4f7fd146bf8aaf8a059ad6cb14" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36433 } }
I20260620 20:27:08.490383 18962 raft_consensus.cc:374] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:08.490509 18962 raft_consensus.cc:729] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c03af821d3e49b5b80e3936c3b8b484, State: Initialized, Role: FOLLOWER
I20260620 20:27:08.490559 18964 consensus_queue.cc:260] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c03af821d3e49b5b80e3936c3b8b484" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41597 } } peers { permanent_uuid: "e20fd9154e1747669f517301541e189b" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38991 } } peers { permanent_uuid: "7520fa4f7fd146bf8aaf8a059ad6cb14" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36433 } }
I20260620 20:27:08.491046 18962 consensus_queue.cc:260] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c03af821d3e49b5b80e3936c3b8b484" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41597 } } peers { permanent_uuid: "e20fd9154e1747669f517301541e189b" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38991 } } peers { permanent_uuid: "7520fa4f7fd146bf8aaf8a059ad6cb14" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36433 } }
I20260620 20:27:08.491822 18943 heartbeater.cc:503] Master 127.31.235.190:35041 was elected leader, sending a full tablet report...
I20260620 20:27:08.494813 18964 ts_tablet_manager.cc:1428] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260620 20:27:08.496307 18962 ts_tablet_manager.cc:1428] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484: Time spent starting tablet: real 0.017s user 0.009s sys 0.006s
W20260620 20:27:08.556921 18676 tablet.cc:2369] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:27:08.588228 18810 tablet.cc:2369] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:27:08.601689 18944 tablet.cc:2369] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:27:08.781737 18969 raft_consensus.cc:482] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:27:08.782078 18969 raft_consensus.cc:504] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c03af821d3e49b5b80e3936c3b8b484" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41597 } } peers { permanent_uuid: "e20fd9154e1747669f517301541e189b" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38991 } } peers { permanent_uuid: "7520fa4f7fd146bf8aaf8a059ad6cb14" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36433 } }
I20260620 20:27:08.783879 18969 leader_election.cc:288] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e20fd9154e1747669f517301541e189b (127.31.235.130:38991), 7520fa4f7fd146bf8aaf8a059ad6cb14 (127.31.235.131:36433)
I20260620 20:27:08.791437 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:08.784576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:55359 (local address 127.31.235.131:36433)
0620 20:27:08.784851 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:08.784860 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:08.785694 (+ 834us) server_negotiation.cc:407] Connection header received
0620 20:27:08.785944 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:08.785950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:08.786024 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:08.786119 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:08.786908 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.787666 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.789651 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.790402 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.790603 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:08.790684 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:08.791029 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 20:27:08.791157 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 20:27:08.791536 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:08.784311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:43981 (local address 127.31.235.130:38991)
0620 20:27:08.784554 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:08.784562 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:08.785363 (+ 801us) server_negotiation.cc:407] Connection header received
0620 20:27:08.785746 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:08.785753 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:08.785830 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:08.785902 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:08.786757 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.787748 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.789851 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.790611 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.790801 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:08.790888 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:08.791150 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 20:27:08.791279 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 20:27:08.791817 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:08.784456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:36433 (local address 127.31.235.129:55359)
0620 20:27:08.785586 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:08.785606 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:08.785808 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:08.786249 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:08.786253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:08.786289 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:08.786722 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:08.786735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.787807 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.787814 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:08.789466 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:08.789479 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.790643 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.790650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:08.790819 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:08.790843 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 20:27:08.791003 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 20:27:08.791140 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":70,"thread_start_us":647,"threads_started":1}
I20260620 20:27:08.791972 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:08.784401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:38991 (local address 127.31.235.129:43981)
0620 20:27:08.785238 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:08.785260 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:08.785555 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:08.786115 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:08.786119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:08.786140 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:08.786584 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:08.786597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.787914 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.787918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:08.789695 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:08.789708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.790837 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.790843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:08.790992 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:08.791014 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 20:27:08.791115 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 20:27:08.791251 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":48,"thread_start_us":425,"threads_started":1}
I20260620 20:27:08.793047 18898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a012a6429e949109ba4806330aca1bf" candidate_uuid: "4c03af821d3e49b5b80e3936c3b8b484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7520fa4f7fd146bf8aaf8a059ad6cb14" is_pre_election: true
I20260620 20:27:08.793047 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a012a6429e949109ba4806330aca1bf" candidate_uuid: "4c03af821d3e49b5b80e3936c3b8b484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e20fd9154e1747669f517301541e189b" is_pre_election: true
I20260620 20:27:08.793650 18898 raft_consensus.cc:2454] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c03af821d3e49b5b80e3936c3b8b484 in term 0.
I20260620 20:27:08.793650 18764 raft_consensus.cc:2454] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c03af821d3e49b5b80e3936c3b8b484 in term 0.
I20260620 20:27:08.794430 18563 leader_election.cc:302] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c03af821d3e49b5b80e3936c3b8b484, 7520fa4f7fd146bf8aaf8a059ad6cb14; no voters:
I20260620 20:27:08.794848 18969 raft_consensus.cc:2781] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:27:08.795032 18969 raft_consensus.cc:482] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:27:08.795156 18969 raft_consensus.cc:3037] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:08.797331 18969 raft_consensus.cc:504] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c03af821d3e49b5b80e3936c3b8b484" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41597 } } peers { permanent_uuid: "e20fd9154e1747669f517301541e189b" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38991 } } peers { permanent_uuid: "7520fa4f7fd146bf8aaf8a059ad6cb14" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36433 } }
I20260620 20:27:08.798100 18969 leader_election.cc:288] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 [CANDIDATE]: Term 1 election: Requested vote from peers e20fd9154e1747669f517301541e189b (127.31.235.130:38991), 7520fa4f7fd146bf8aaf8a059ad6cb14 (127.31.235.131:36433)
I20260620 20:27:08.798612 18764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a012a6429e949109ba4806330aca1bf" candidate_uuid: "4c03af821d3e49b5b80e3936c3b8b484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e20fd9154e1747669f517301541e189b"
I20260620 20:27:08.798651 18898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a012a6429e949109ba4806330aca1bf" candidate_uuid: "4c03af821d3e49b5b80e3936c3b8b484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7520fa4f7fd146bf8aaf8a059ad6cb14"
I20260620 20:27:08.798806 18764 raft_consensus.cc:3037] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:08.798885 18898 raft_consensus.cc:3037] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:08.801002 18764 raft_consensus.cc:2454] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c03af821d3e49b5b80e3936c3b8b484 in term 1.
I20260620 20:27:08.801208 18898 raft_consensus.cc:2454] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c03af821d3e49b5b80e3936c3b8b484 in term 1.
I20260620 20:27:08.801469 18564 leader_election.cc:302] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c03af821d3e49b5b80e3936c3b8b484, e20fd9154e1747669f517301541e189b; no voters:
I20260620 20:27:08.801810 18969 raft_consensus.cc:2781] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:08.802268 18969 raft_consensus.cc:686] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 [term 1 LEADER]: Becoming Leader. State: Replica: 4c03af821d3e49b5b80e3936c3b8b484, State: Running, Role: LEADER
I20260620 20:27:08.802780 18969 consensus_queue.cc:237] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c03af821d3e49b5b80e3936c3b8b484" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41597 } } peers { permanent_uuid: "e20fd9154e1747669f517301541e189b" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38991 } } peers { permanent_uuid: "7520fa4f7fd146bf8aaf8a059ad6cb14" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36433 } }
I20260620 20:27:08.807730 18484 catalog_manager.cc:5495] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c03af821d3e49b5b80e3936c3b8b484 (127.31.235.129). New cstate: current_term: 1 leader_uuid: "4c03af821d3e49b5b80e3936c3b8b484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c03af821d3e49b5b80e3936c3b8b484" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e20fd9154e1747669f517301541e189b" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7520fa4f7fd146bf8aaf8a059ad6cb14" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36433 } health_report { overall_health: UNKNOWN } } }
I20260620 20:27:08.912508 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:08.906346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41396 (local address 127.31.235.129:41597)
0620 20:27:08.906550 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:08.906554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:08.907087 (+ 533us) server_negotiation.cc:407] Connection header received
0620 20:27:08.907260 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:08.907264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:08.907346 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:08.907479 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:27:08.907946 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.908668 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.910241 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.910648 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.910783 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:08.911381 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:27:08.911887 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:27:08.912096 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:08.912209 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 20:27:08.912257 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:27:08.921259 18764 raft_consensus.cc:1261] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b [term 1 FOLLOWER]: Refusing update from remote peer 4c03af821d3e49b5b80e3936c3b8b484: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:27:08.921249 18898 raft_consensus.cc:1261] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14 [term 1 FOLLOWER]: Refusing update from remote peer 4c03af821d3e49b5b80e3936c3b8b484: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:27:08.922483 18972 consensus_queue.cc:1035] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7520fa4f7fd146bf8aaf8a059ad6cb14" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 36433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:27:08.922883 18969 consensus_queue.cc:1035] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e20fd9154e1747669f517301541e189b" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 38991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:27:08.934549 18976 mvcc.cc:205] Tried to move back new op lower bound from 7299019689650425856 to 7299019689180020736. Current Snapshot: MvccSnapshot[applied={T|T < 7299019689650425856}]
I20260620 20:27:08.937783 18977 mvcc.cc:205] Tried to move back new op lower bound from 7299019689650425856 to 7299019689180020736. Current Snapshot: MvccSnapshot[applied={T|T < 7299019689650425856}]
I20260620 20:27:08.949664 18485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:27:08.949931 18485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:27:08.953637 18485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c994e05dc5e4eb3a44bc460e5a587c8: Sending DeleteTablet for 3 replicas of tablet 0a012a6429e949109ba4806330aca1bf
I20260620 20:27:08.954793 18610 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a012a6429e949109ba4806330aca1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:27:08 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:41384
I20260620 20:27:08.955003 18744 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a012a6429e949109ba4806330aca1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:27:08 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:32846
I20260620 20:27:08.955003 18878 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a012a6429e949109ba4806330aca1bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:27:08 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:36362
I20260620 20:27:08.956297 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 18547
I20260620 20:27:08.966805 18987 tablet_replica.cc:330] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14: stopping tablet replica
I20260620 20:27:08.967448 18987 raft_consensus.cc:2229] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:27:08.968137 18987 raft_consensus.cc:2258] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:27:08.969025 18986 tablet_replica.cc:330] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b: stopping tablet replica
I20260620 20:27:08.969684 18985 tablet_replica.cc:330] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484: stopping tablet replica
I20260620 20:27:08.970201 18986 raft_consensus.cc:2229] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:27:08.970623 18987 ts_tablet_manager.cc:1905] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:27:08.970690 18985 raft_consensus.cc:2229] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:27:08.971531 18985 raft_consensus.cc:2258] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:27:08.972126 18986 raft_consensus.cc:2258] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:27:08.976836 18986 ts_tablet_manager.cc:1905] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 20:27:08 dist-test-slave-zzwf krb5kdc[18441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:27:08.979545 18987 ts_tablet_manager.cc:1918] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:27:08.979744 18987 log.cc:1196] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/0a012a6429e949109ba4806330aca1bf
I20260620 20:27:08.980090 18985 ts_tablet_manager.cc:1905] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:27:08.980336 18987 ts_tablet_manager.cc:1939] T 0a012a6429e949109ba4806330aca1bf P 7520fa4f7fd146bf8aaf8a059ad6cb14: Deleting consensus metadata
I20260620 20:27:08.982430 18470 catalog_manager.cc:4841] TS 7520fa4f7fd146bf8aaf8a059ad6cb14 (127.31.235.131:36433): tablet 0a012a6429e949109ba4806330aca1bf (table test-table [id=4f53cae3662e418eabed843523f2f4e3]) successfully deleted
I20260620 20:27:08.987839 18986 ts_tablet_manager.cc:1918] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:27:08.987890 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:08.957667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41398 (local address 127.31.235.129:41597)
0620 20:27:08.957830 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:08.957836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:08.966410 (+ 8574us) server_negotiation.cc:407] Connection header received
0620 20:27:08.967378 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:08.967384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:08.967495 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:08.967588 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:08.969740 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.970996 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.972937 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:08.973683 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:08.973790 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:08.979656 (+ 5866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:08.979674 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:08.979692 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:08.979744 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:08.984111 (+ 4367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:08.985053 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:08.985065 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:08.985072 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:08.985257 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:08.985798 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:08.985804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:08.985807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:08.986275 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:08.986585 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:08.987217 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 20:27:08.987490 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:27:08.988018 18986 log.cc:1196] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/0a012a6429e949109ba4806330aca1bf
I20260620 20:27:08.988619 18986 ts_tablet_manager.cc:1939] T 0a012a6429e949109ba4806330aca1bf P e20fd9154e1747669f517301541e189b: Deleting consensus metadata
I20260620 20:27:08.989357 18670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:08.990628 18471 catalog_manager.cc:4841] TS e20fd9154e1747669f517301541e189b (127.31.235.130:38991): tablet 0a012a6429e949109ba4806330aca1bf (table test-table [id=4f53cae3662e418eabed843523f2f4e3]) successfully deleted
I20260620 20:27:09.088855 18985 ts_tablet_manager.cc:1918] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:27:09.089182 18985 log.cc:1196] T 0a012a6429e949109ba4806330aca1bf P 4c03af821d3e49b5b80e3936c3b8b484: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/0a012a6429e949109ba4806330aca1bf
I20260620 20:27:09.089627 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 18547
W20260620 20:27:09.108646 18472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.235.129:41597 (error 108)
I20260620 20:27:09.108841 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 18681
I20260620 20:27:09.110868 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:09.110116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:41597 (local address 127.0.0.1:41406)
0620 20:27:09.110586 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:09.110698 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":249,"threads_started":1}
W20260620 20:27:09.111083 18472 catalog_manager.cc:4584] TS 4c03af821d3e49b5b80e3936c3b8b484 (127.31.235.129:41597): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a012a6429e949109ba4806330aca1bf: Network error: Client connection negotiation failed: client connection to 127.31.235.129:41597: connect: Connection refused (error 111)
Jun 20 20:27:09 dist-test-slave-zzwf krb5kdc[18441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.130@KRBTEST.COM
I20260620 20:27:09.125130 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:09.109768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.31.235.130:38991)
0620 20:27:09.109941 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:09.109949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:09.109969 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:27:09.110169 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:09.110176 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:09.110284 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:09.110400 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:09.112000 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:09.112725 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:09.114184 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:09.114596 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:09.114703 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:09.119298 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:09.119312 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:09.119329 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:09.119370 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:09.122411 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:09.123145 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:09.123153 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:09.123155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:09.123317 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:09.123697 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:09.123700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:09.123702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:09.124086 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:09.124332 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:09.124684 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 20:27:09.124854 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:27:09.125861 18804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:09.164845 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:09.164371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:41597 (local address 127.0.0.1:41418)
0620 20:27:09.164618 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:09.164703 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 20:27:09.166169 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:09.165713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:41597 (local address 127.0.0.1:41434)
0620 20:27:09.165908 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:09.165995 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 20:27:09.218465 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 18681
I20260620 20:27:09.226843 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:09.226401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:41597 (local address 127.0.0.1:41436)
0620 20:27:09.226610 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:09.226692 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 20:27:09.227918 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:09.227530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:41597 (local address 127.0.0.1:41440)
0620 20:27:09.227698 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:09.227792 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 20:27:09.237049 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 18815
Jun 20 20:27:09 dist-test-slave-zzwf krb5kdc[18441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.131@KRBTEST.COM
I20260620 20:27:09.252519 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:09.237878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36366 (local address 127.31.235.131:36433)
0620 20:27:09.238041 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:09.238045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:09.238064 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:27:09.238136 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:09.238140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:09.238204 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:09.238295 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:09.239693 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:09.240409 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:09.241800 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:09.242193 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:09.242330 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:09.247054 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:09.247069 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:09.247085 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:09.247118 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:09.249856 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:09.250569 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:09.250577 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:09.250580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:09.250747 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:09.251092 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:09.251095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:09.251097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:09.251502 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:09.251763 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:09.252033 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 20:27:09.252220 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 20:27:09.253209 18938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:09.309118 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:09.308612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:41597 (local address 127.0.0.1:41450)
0620 20:27:09.308852 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:09.308947 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 20:27:09.310282 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:09.309862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:41597 (local address 127.0.0.1:41452)
0620 20:27:09.310073 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:09.310141 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 20:27:09.342336 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 18815
I20260620 20:27:09.359501 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 18454
I20260620 20:27:09.360638 18516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:09.472829 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 18454
2026-06-20T20:27:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4678 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T20:27:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:09Z Disabled control of system clock
I20260620 20:27:09.551348 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:38661
--webserver_interface=127.31.235.190
--webserver_port=0
--builtin_ntp_servers=127.31.235.148:36127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:38661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:27:09.893330 18998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:09.893718 18998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:09.893873 18998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:09.901535 18998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:09.901677 18998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:09.901739 18998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:09.901824 18998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:09.901913 18998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:09.911504 18998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:36127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:38661
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:38661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:09.912855 18998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:09.914721 18998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:09.927340 19003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:09.927421 19004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:09.927435 19006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:09.931121 18998 server_base.cc:1034] running on GCE node
I20260620 20:27:09.932027 18998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:09.933661 18998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:09.934900 18998 hybrid_clock.cc:648] HybridClock initialized: now 1781987229934851 us; error 36 us; skew 500 ppm
I20260620 20:27:09.935379 18998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:27:09.938030 18998 webserver.cc:460] Webserver started at http://127.31.235.190:33043/ using document root <none> and password file <none>
I20260620 20:27:09.938793 18998 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:09.938894 18998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:09.939245 18998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:09.942440 18998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "157780d97ac846af9352704822a81d21"
format_stamp: "Formatted at 2026-06-20 20:27:09 on dist-test-slave-zzwf"
server_key: "a330c1b0956957235edbce40a0fefe99"
server_key_iv: "1d29a8be99886fa2f7b67cb19f15a51a"
server_key_version: "encryptionkey@0"
I20260620 20:27:09.943364 18998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "157780d97ac846af9352704822a81d21"
format_stamp: "Formatted at 2026-06-20 20:27:09 on dist-test-slave-zzwf"
server_key: "a330c1b0956957235edbce40a0fefe99"
server_key_iv: "1d29a8be99886fa2f7b67cb19f15a51a"
server_key_version: "encryptionkey@0"
I20260620 20:27:09.949043 18998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:27:09.952752 19012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:09.954377 18998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:27:09.954583 18998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "157780d97ac846af9352704822a81d21"
format_stamp: "Formatted at 2026-06-20 20:27:09 on dist-test-slave-zzwf"
server_key: "a330c1b0956957235edbce40a0fefe99"
server_key_iv: "1d29a8be99886fa2f7b67cb19f15a51a"
server_key_version: "encryptionkey@0"
I20260620 20:27:09.954839 18998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:09.974953 18998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:09.975874 18998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:09.976188 18998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:09.995630 18998 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:38661
I20260620 20:27:09.995656 19063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:38661 every 8 connection(s)
I20260620 20:27:09.997219 18998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:10.001161 19064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:10.006006 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 18998
I20260620 20:27:10.006184 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:10.006762 32686 external_mini_cluster.cc:1428] Setting key 891aeb9abf437d0974f1e46a8ad4d4b3
I20260620 20:27:10.012667 19064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21: Bootstrap starting.
I20260620 20:27:10.015807 19064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:10.016860 19064 log.cc:824] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:10.020314 19064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21: No bootstrap required, opened a new log
I20260620 20:27:10.020763 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:10.008739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38320 (local address 127.31.235.190:38661)
0620 20:27:10.009848 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:10.009859 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:10.009902 (+ 43us) server_negotiation.cc:407] Connection header received
0620 20:27:10.011606 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:10.011629 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:10.012074 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:10.012458 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:10.013017 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:10.014382 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:10.015762 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:10.016529 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:10.016742 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:10.018014 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:10.018025 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:10.018038 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:10.018714 (+ 676us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:10.018830 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:10.018856 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:10.018974 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:10.019181 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 20:27:10.019510 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":391,"threads_started":1}
I20260620 20:27:10.028286 19064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157780d97ac846af9352704822a81d21" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 38661 } }
I20260620 20:27:10.028646 19064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:10.028721 19064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157780d97ac846af9352704822a81d21, State: Initialized, Role: FOLLOWER
I20260620 20:27:10.029377 19064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157780d97ac846af9352704822a81d21" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 38661 } }
I20260620 20:27:10.029695 19064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:10.029850 19064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:10.030047 19064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:10.032239 19064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157780d97ac846af9352704822a81d21" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 38661 } }
I20260620 20:27:10.032766 19064 leader_election.cc:302] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 157780d97ac846af9352704822a81d21; no voters:
I20260620 20:27:10.033398 19064 leader_election.cc:288] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:10.033619 19069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:10.034827 19069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21 [term 1 LEADER]: Becoming Leader. State: Replica: 157780d97ac846af9352704822a81d21, State: Running, Role: LEADER
I20260620 20:27:10.035331 19069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157780d97ac846af9352704822a81d21" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 38661 } }
I20260620 20:27:10.035609 19064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:10.038318 19070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "157780d97ac846af9352704822a81d21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157780d97ac846af9352704822a81d21" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 38661 } } }
I20260620 20:27:10.038373 19071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 157780d97ac846af9352704822a81d21. Latest consensus state: current_term: 1 leader_uuid: "157780d97ac846af9352704822a81d21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "157780d97ac846af9352704822a81d21" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 38661 } } }
I20260620 20:27:10.039129 19070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:10.039127 19071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:10.040709 19077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:10.045797 19077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:10.055954 19077 catalog_manager.cc:1347] Generated new cluster ID: b5deeea44da840d3b82880085d4e558b
I20260620 20:27:10.056084 19077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:10.097936 19077 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:10.099223 19077 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:10.112085 19077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21: Generated new TSK 0
I20260620 20:27:10.112991 19077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:10.125121 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:38661
--builtin_ntp_servers=127.31.235.148:36127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:27:10.482208 19088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:10.482473 19088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:10.482558 19088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:10.489760 19088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:10.489859 19088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:10.489971 19088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:27:10.499022 19088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:36127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:38661
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:10.500233 19088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:10.501940 19088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:10.515205 19093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:10.515208 19094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:10.515434 19096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:10.515960 19088 server_base.cc:1034] running on GCE node
I20260620 20:27:10.516642 19088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:10.517570 19088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:10.518802 19088 hybrid_clock.cc:648] HybridClock initialized: now 1781987230518760 us; error 53 us; skew 500 ppm
I20260620 20:27:10.519133 19088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:27:10.521267 19088 webserver.cc:460] Webserver started at http://127.31.235.129:46309/ using document root <none> and password file <none>
I20260620 20:27:10.522048 19088 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:10.522176 19088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:10.522490 19088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:10.525250 19088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "bb3c93ab3a934d559efe4453b6da5868"
format_stamp: "Formatted at 2026-06-20 20:27:10 on dist-test-slave-zzwf"
server_key: "690494e4916b44a2b4451b4b5a7b6f6b"
server_key_iv: "82b01f5f54a34f2c92fdcc39292d942d"
server_key_version: "encryptionkey@0"
I20260620 20:27:10.526072 19088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "bb3c93ab3a934d559efe4453b6da5868"
format_stamp: "Formatted at 2026-06-20 20:27:10 on dist-test-slave-zzwf"
server_key: "690494e4916b44a2b4451b4b5a7b6f6b"
server_key_iv: "82b01f5f54a34f2c92fdcc39292d942d"
server_key_version: "encryptionkey@0"
I20260620 20:27:10.531038 19088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:27:10.534569 19102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:10.535845 19088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:27:10.536084 19088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "bb3c93ab3a934d559efe4453b6da5868"
format_stamp: "Formatted at 2026-06-20 20:27:10 on dist-test-slave-zzwf"
server_key: "690494e4916b44a2b4451b4b5a7b6f6b"
server_key_iv: "82b01f5f54a34f2c92fdcc39292d942d"
server_key_version: "encryptionkey@0"
I20260620 20:27:10.536327 19088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:10.553898 19088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:10.554762 19088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:10.555115 19088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:10.556106 19088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:27:10.557725 19088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:27:10.557860 19088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:10.557988 19088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:27:10.558075 19088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:10.591022 19088 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:45519
I20260620 20:27:10.591077 19214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:45519 every 8 connection(s)
I20260620 20:27:10.592305 19088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:27:10.593919 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 19088
I20260620 20:27:10.594101 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:27:10.594636 32686 external_mini_cluster.cc:1428] Setting key 432ebecebb416e889e6f316170514541
I20260620 20:27:10.597731 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:38661
--builtin_ntp_servers=127.31.235.148:36127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:27:10.605157 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:10.595023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:43573 (local address 127.31.235.190:38661)
0620 20:27:10.595541 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:10.595547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:10.596845 (+ 1298us) server_negotiation.cc:407] Connection header received
0620 20:27:10.598793 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:10.598801 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:10.598888 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:10.599008 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:10.600038 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:10.600775 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:10.602389 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:10.602779 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:10.602921 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:10.604290 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:10.604295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:10.604298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:10.604315 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:10.604353 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:10.604365 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:10.604468 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:10.604691 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 20:27:10.604789 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":277,"threads_started":1}
I20260620 20:27:10.606181 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:10.595684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:38661 (local address 127.31.235.129:43573)
0620 20:27:10.596706 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:10.596742 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:10.598528 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:10.599236 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:10.599244 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:10.599329 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:10.599873 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:10.599890 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:10.600938 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:10.600944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:10.602230 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:10.602242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:10.603012 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:10.603022 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:10.603251 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:10.603972 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:27:10.603975 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:10.604020 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:27:10.604099 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:10.604468 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:10.604482 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:10.604492 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 20:27:10.604681 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 20:27:10.605073 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":301,"threads_started":1}
I20260620 20:27:10.607725 19215 heartbeater.cc:347] Connected to a master server at 127.31.235.190:38661
I20260620 20:27:10.608032 19215 heartbeater.cc:464] Registering TS with master...
I20260620 20:27:10.608757 19215 heartbeater.cc:511] Master 127.31.235.190:38661 requested a full tablet report, sending...
I20260620 20:27:10.610776 19029 ts_manager.cc:194] Registered new tserver with Master: bb3c93ab3a934d559efe4453b6da5868 (127.31.235.129:45519)
I20260620 20:27:10.612452 19029 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.235.129:43573
W20260620 20:27:10.922622 19219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:10.922950 19219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:10.923107 19219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:10.930411 19219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:10.930548 19219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:10.930720 19219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:27:10.940115 19219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:36127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:38661
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:10.941648 19219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:10.943542 19219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:10.956965 19227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:10.957048 19224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:10.957074 19225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:10.957726 19219 server_base.cc:1034] running on GCE node
I20260620 20:27:10.958324 19219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:10.959355 19219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:10.960582 19219 hybrid_clock.cc:648] HybridClock initialized: now 1781987230960524 us; error 75 us; skew 500 ppm
I20260620 20:27:10.960904 19219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:27:10.963246 19219 webserver.cc:460] Webserver started at http://127.31.235.130:42435/ using document root <none> and password file <none>
I20260620 20:27:10.963948 19219 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:10.964082 19219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:10.964406 19219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:10.967320 19219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "8a9d392805384d32b2c9ccff9eb973a1"
format_stamp: "Formatted at 2026-06-20 20:27:10 on dist-test-slave-zzwf"
server_key: "a67035b255060697606432e73ef7b955"
server_key_iv: "c9e04b7a60c22d572cc8a2af85b94acc"
server_key_version: "encryptionkey@0"
I20260620 20:27:10.968122 19219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "8a9d392805384d32b2c9ccff9eb973a1"
format_stamp: "Formatted at 2026-06-20 20:27:10 on dist-test-slave-zzwf"
server_key: "a67035b255060697606432e73ef7b955"
server_key_iv: "c9e04b7a60c22d572cc8a2af85b94acc"
server_key_version: "encryptionkey@0"
I20260620 20:27:10.973141 19219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:27:10.976715 19233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:10.978096 19219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:27:10.978276 19219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "8a9d392805384d32b2c9ccff9eb973a1"
format_stamp: "Formatted at 2026-06-20 20:27:10 on dist-test-slave-zzwf"
server_key: "a67035b255060697606432e73ef7b955"
server_key_iv: "c9e04b7a60c22d572cc8a2af85b94acc"
server_key_version: "encryptionkey@0"
I20260620 20:27:10.978502 19219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:10.993096 19219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:10.993992 19219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:10.994297 19219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:10.995368 19219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:27:10.996865 19219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:27:10.996950 19219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:10.997102 19219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:27:10.997177 19219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:11.031826 19219 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:36627
I20260620 20:27:11.031876 19345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:36627 every 8 connection(s)
I20260620 20:27:11.033151 19219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:27:11.042982 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 19219
I20260620 20:27:11.043174 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:27:11.043627 32686 external_mini_cluster.cc:1428] Setting key 8c5a1f987f2c2cbd4a4e18cd14dd937f
I20260620 20:27:11.046450 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:11.035776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:46501 (local address 127.31.235.190:38661)
0620 20:27:11.036020 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:11.036025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:11.037372 (+ 1347us) server_negotiation.cc:407] Connection header received
0620 20:27:11.039193 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:11.039198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:11.039266 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:11.039368 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:11.040531 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.041368 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.043071 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.043587 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.043732 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:11.045363 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:11.045370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:11.045374 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:11.045395 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:11.045434 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:11.045449 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:11.045634 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:11.045925 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 20:27:11.046037 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 20:27:11.047188 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:38661
--builtin_ntp_servers=127.31.235.148:36127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:27:11.047439 19348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:11.036257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:38661 (local address 127.31.235.130:46501)
0620 20:27:11.037248 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:11.037283 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:11.038933 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:11.039638 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:11.039646 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:11.039742 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:11.040366 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:11.040383 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.041571 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.041576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:11.042917 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:11.042928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.043870 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.043878 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:11.044119 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:11.044966 (+ 847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:27:11.044970 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:11.045022 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:27:11.045121 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:11.045640 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:11.045651 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:11.045659 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 20:27:11.045876 (+ 217us) client_negotiation.cc:240] Negotiation successful
0620 20:27:11.046193 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":380,"threads_started":1}
I20260620 20:27:11.050765 19346 heartbeater.cc:347] Connected to a master server at 127.31.235.190:38661
I20260620 20:27:11.051110 19346 heartbeater.cc:464] Registering TS with master...
I20260620 20:27:11.051864 19346 heartbeater.cc:511] Master 127.31.235.190:38661 requested a full tablet report, sending...
I20260620 20:27:11.053354 19029 ts_manager.cc:194] Registered new tserver with Master: 8a9d392805384d32b2c9ccff9eb973a1 (127.31.235.130:36627)
I20260620 20:27:11.054201 19029 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.235.130:46501
W20260620 20:27:11.376055 19349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:11.376325 19349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:11.376483 19349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:11.383808 19349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:11.383922 19349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:11.384120 19349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:27:11.393994 19349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:36127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:38661
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:11.395295 19349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:11.397017 19349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:11.410072 19354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:11.410109 19357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:11.410037 19355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:11.410861 19349 server_base.cc:1034] running on GCE node
I20260620 20:27:11.411453 19349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:11.412377 19349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:11.413597 19349 hybrid_clock.cc:648] HybridClock initialized: now 1781987231413542 us; error 57 us; skew 500 ppm
I20260620 20:27:11.413914 19349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:27:11.416271 19349 webserver.cc:460] Webserver started at http://127.31.235.131:34697/ using document root <none> and password file <none>
I20260620 20:27:11.416930 19349 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:11.417052 19349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:11.417366 19349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:11.420135 19349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "2ccf072073144251a1fa67e10bb3a3b2"
format_stamp: "Formatted at 2026-06-20 20:27:11 on dist-test-slave-zzwf"
server_key: "6daddc6b4036b37cf67051208139d90b"
server_key_iv: "9447bc56212398155b1ac689838ad199"
server_key_version: "encryptionkey@0"
I20260620 20:27:11.420957 19349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "2ccf072073144251a1fa67e10bb3a3b2"
format_stamp: "Formatted at 2026-06-20 20:27:11 on dist-test-slave-zzwf"
server_key: "6daddc6b4036b37cf67051208139d90b"
server_key_iv: "9447bc56212398155b1ac689838ad199"
server_key_version: "encryptionkey@0"
I20260620 20:27:11.426000 19349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:27:11.429320 19363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:11.430644 19349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:27:11.430817 19349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "2ccf072073144251a1fa67e10bb3a3b2"
format_stamp: "Formatted at 2026-06-20 20:27:11 on dist-test-slave-zzwf"
server_key: "6daddc6b4036b37cf67051208139d90b"
server_key_iv: "9447bc56212398155b1ac689838ad199"
server_key_version: "encryptionkey@0"
I20260620 20:27:11.431036 19349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:11.461814 19349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:11.462677 19349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:11.462994 19349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:11.463982 19349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:27:11.465483 19349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:27:11.465622 19349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:11.465759 19349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:27:11.465852 19349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:11.497116 19349 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:35149
I20260620 20:27:11.497208 19475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:35149 every 8 connection(s)
I20260620 20:27:11.498413 19349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:27:11.503957 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 19349
I20260620 20:27:11.504117 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:27:11.504684 32686 external_mini_cluster.cc:1428] Setting key 4787f6416a1c9956dc5a7b0aab13f321
I20260620 20:27:11.510910 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:11.500906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:35909 (local address 127.31.235.190:38661)
0620 20:27:11.501087 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:11.501091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:11.502483 (+ 1392us) server_negotiation.cc:407] Connection header received
0620 20:27:11.504308 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:11.504314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:11.504411 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:11.504526 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:11.505729 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.506737 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.508301 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.508698 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.508831 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:11.510098 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:11.510104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:11.510109 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:11.510135 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:11.510165 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:11.510176 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:11.510271 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:11.510500 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 20:27:11.510590 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 20:27:11.511801 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:11.501427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:38661 (local address 127.31.235.131:35909)
0620 20:27:11.502329 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:11.502362 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:11.503960 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:11.504703 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:11.504710 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:11.504838 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:11.505566 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:11.505585 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.506885 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.506890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:11.508159 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:11.508170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.508919 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.508925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:11.509147 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:11.509793 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:27:11.509796 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:11.509832 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:27:11.509910 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:11.510270 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:11.510282 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:11.510292 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 20:27:11.510465 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 20:27:11.510815 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":265,"threads_started":1}
I20260620 20:27:11.512953 19476 heartbeater.cc:347] Connected to a master server at 127.31.235.190:38661
I20260620 20:27:11.513233 19476 heartbeater.cc:464] Registering TS with master...
I20260620 20:27:11.513974 19476 heartbeater.cc:511] Master 127.31.235.190:38661 requested a full tablet report, sending...
I20260620 20:27:11.515347 19029 ts_manager.cc:194] Registered new tserver with Master: 2ccf072073144251a1fa67e10bb3a3b2 (127.31.235.131:35149)
I20260620 20:27:11.516184 19029 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.235.131:35909
I20260620 20:27:11.519461 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:27:11.529138 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:11.523354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38324 (local address 127.31.235.190:38661)
0620 20:27:11.523541 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:11.523548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:11.524026 (+ 478us) server_negotiation.cc:407] Connection header received
0620 20:27:11.524187 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:11.524191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:11.524260 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:11.524352 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:11.524758 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.525443 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.526941 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.527356 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.527458 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:11.528517 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:11.528522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:11.528525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:11.528541 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:11.528570 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:11.528581 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:11.528688 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:11.528870 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 20:27:11.528916 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:27:11.533025 19029 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:27:11.540315 19029 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:27:11.575279 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:11.552932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52554 (local address 127.31.235.130:36627)
0620 20:27:11.565427 (+ 12495us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:11.565438 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:11.566452 (+ 1014us) server_negotiation.cc:407] Connection header received
0620 20:27:11.566780 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:11.566789 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:11.567066 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:11.567272 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:11.568423 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.569883 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.572800 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.574070 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.574241 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:11.574478 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:11.574734 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 20:27:11.574897 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12366,"thread_start_us":12280,"threads_started":1}
I20260620 20:27:11.575361 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:11.552669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:36627 (local address 127.0.0.1:52554)
0620 20:27:11.566212 (+ 13543us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:11.566279 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:11.566521 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:11.567490 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:11.567498 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:11.567798 (+ 300us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:11.568240 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:11.568251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.570047 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.570054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:11.572545 (+ 2491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:11.572565 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.574262 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.574268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:11.574458 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:11.574509 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 20:27:11.574686 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 20:27:11.574879 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13178,"mutex_wait_us":2,"spinlock_wait_cycles":449664,"thread_start_us":13646,"threads_started":1}
I20260620 20:27:11.575420 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:11.566958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:45519 (local address 127.0.0.1:53180)
0620 20:27:11.567483 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:11.567498 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:11.567791 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:11.568772 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:11.568779 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:11.568808 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:11.569332 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:11.569345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.571231 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.571237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:11.572891 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:11.572906 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.574540 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.574547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:11.574765 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:11.574800 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 20:27:11.574866 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 20:27:11.575003 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":92,"spinlock_wait_cycles":413824,"thread_start_us":288,"threads_started":1}
I20260620 20:27:11.576478 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:11.553321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.31.235.131:35149)
0620 20:27:11.565186 (+ 11865us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:11.565197 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:11.566808 (+ 1611us) server_negotiation.cc:407] Connection header received
0620 20:27:11.567943 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:11.567953 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:11.568172 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:11.568328 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:11.569328 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.570964 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.573622 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.574995 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.575170 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:11.575382 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:11.575916 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 20:27:11.576113 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11671,"thread_start_us":11499,"threads_started":1}
I20260620 20:27:11.576776 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:11.567041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53180 (local address 127.31.235.129:45519)
0620 20:27:11.568224 (+ 1183us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:11.568232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:11.568260 (+ 28us) server_negotiation.cc:407] Connection header received
0620 20:27:11.568348 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:11.568354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:11.568576 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:11.568762 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:11.569551 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.571033 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.573075 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.574326 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.575993 (+ 1667us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:11.576188 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:11.576254 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 20:27:11.576406 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1017,"thread_start_us":902,"threads_started":1}
I20260620 20:27:11.577180 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:11.553019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:35149 (local address 127.0.0.1:47244)
0620 20:27:11.566695 (+ 13676us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:11.566712 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:11.567737 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:11.568622 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:11.568626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:11.568650 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:11.569153 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:11.569164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.571106 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.571111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:11.573431 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:11.573446 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.575515 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.575523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:11.575703 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:11.575747 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 20:27:11.576734 (+ 987us) client_negotiation.cc:240] Negotiation successful
0620 20:27:11.576874 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13525,"mutex_wait_us":150,"thread_start_us":12642,"threads_started":1}
I20260620 20:27:11.580031 19150 tablet_service.cc:1467] Processing CreateTablet for tablet 3d10d7bf748c4c84a42070b777e1eed1 (DEFAULT_TABLE table=test-table [id=171c3f90503e43058a3bdde9bb490b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:27:11.580377 19411 tablet_service.cc:1467] Processing CreateTablet for tablet 3d10d7bf748c4c84a42070b777e1eed1 (DEFAULT_TABLE table=test-table [id=171c3f90503e43058a3bdde9bb490b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:27:11.580356 19281 tablet_service.cc:1467] Processing CreateTablet for tablet 3d10d7bf748c4c84a42070b777e1eed1 (DEFAULT_TABLE table=test-table [id=171c3f90503e43058a3bdde9bb490b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:27:11.581327 19150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d10d7bf748c4c84a42070b777e1eed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:11.581698 19281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d10d7bf748c4c84a42070b777e1eed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:11.581698 19411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d10d7bf748c4c84a42070b777e1eed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:11.599987 19494 tablet_bootstrap.cc:492] T 3d10d7bf748c4c84a42070b777e1eed1 P bb3c93ab3a934d559efe4453b6da5868: Bootstrap starting.
I20260620 20:27:11.601161 19495 tablet_bootstrap.cc:492] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1: Bootstrap starting.
I20260620 20:27:11.601966 19496 tablet_bootstrap.cc:492] T 3d10d7bf748c4c84a42070b777e1eed1 P 2ccf072073144251a1fa67e10bb3a3b2: Bootstrap starting.
I20260620 20:27:11.605247 19494 tablet_bootstrap.cc:654] T 3d10d7bf748c4c84a42070b777e1eed1 P bb3c93ab3a934d559efe4453b6da5868: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:11.605597 19495 tablet_bootstrap.cc:654] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:11.606550 19494 log.cc:824] T 3d10d7bf748c4c84a42070b777e1eed1 P bb3c93ab3a934d559efe4453b6da5868: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:11.606828 19495 log.cc:824] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:11.607270 19496 tablet_bootstrap.cc:654] T 3d10d7bf748c4c84a42070b777e1eed1 P 2ccf072073144251a1fa67e10bb3a3b2: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:11.608480 19496 log.cc:824] T 3d10d7bf748c4c84a42070b777e1eed1 P 2ccf072073144251a1fa67e10bb3a3b2: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:11.609870 19495 tablet_bootstrap.cc:492] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1: No bootstrap required, opened a new log
I20260620 20:27:11.609870 19494 tablet_bootstrap.cc:492] T 3d10d7bf748c4c84a42070b777e1eed1 P bb3c93ab3a934d559efe4453b6da5868: No bootstrap required, opened a new log
I20260620 20:27:11.610148 19495 ts_tablet_manager.cc:1397] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 20:27:11.610146 19494 ts_tablet_manager.cc:1397] T 3d10d7bf748c4c84a42070b777e1eed1 P bb3c93ab3a934d559efe4453b6da5868: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 20:27:11.611135 19496 tablet_bootstrap.cc:492] T 3d10d7bf748c4c84a42070b777e1eed1 P 2ccf072073144251a1fa67e10bb3a3b2: No bootstrap required, opened a new log
I20260620 20:27:11.611397 19496 ts_tablet_manager.cc:1397] T 3d10d7bf748c4c84a42070b777e1eed1 P 2ccf072073144251a1fa67e10bb3a3b2: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 20:27:11.615396 19215 heartbeater.cc:503] Master 127.31.235.190:38661 was elected leader, sending a full tablet report...
I20260620 20:27:11.619746 19496 raft_consensus.cc:348] T 3d10d7bf748c4c84a42070b777e1eed1 P 2ccf072073144251a1fa67e10bb3a3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9d392805384d32b2c9ccff9eb973a1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36627 } } peers { permanent_uuid: "2ccf072073144251a1fa67e10bb3a3b2" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35149 } } peers { permanent_uuid: "bb3c93ab3a934d559efe4453b6da5868" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45519 } }
I20260620 20:27:11.620362 19496 raft_consensus.cc:374] T 3d10d7bf748c4c84a42070b777e1eed1 P 2ccf072073144251a1fa67e10bb3a3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:11.620522 19496 raft_consensus.cc:729] T 3d10d7bf748c4c84a42070b777e1eed1 P 2ccf072073144251a1fa67e10bb3a3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ccf072073144251a1fa67e10bb3a3b2, State: Initialized, Role: FOLLOWER
I20260620 20:27:11.621049 19494 raft_consensus.cc:348] T 3d10d7bf748c4c84a42070b777e1eed1 P bb3c93ab3a934d559efe4453b6da5868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9d392805384d32b2c9ccff9eb973a1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36627 } } peers { permanent_uuid: "2ccf072073144251a1fa67e10bb3a3b2" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35149 } } peers { permanent_uuid: "bb3c93ab3a934d559efe4453b6da5868" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45519 } }
I20260620 20:27:11.621572 19494 raft_consensus.cc:374] T 3d10d7bf748c4c84a42070b777e1eed1 P bb3c93ab3a934d559efe4453b6da5868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:11.621372 19496 consensus_queue.cc:260] T 3d10d7bf748c4c84a42070b777e1eed1 P 2ccf072073144251a1fa67e10bb3a3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9d392805384d32b2c9ccff9eb973a1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36627 } } peers { permanent_uuid: "2ccf072073144251a1fa67e10bb3a3b2" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35149 } } peers { permanent_uuid: "bb3c93ab3a934d559efe4453b6da5868" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45519 } }
I20260620 20:27:11.621372 19495 raft_consensus.cc:348] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9d392805384d32b2c9ccff9eb973a1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36627 } } peers { permanent_uuid: "2ccf072073144251a1fa67e10bb3a3b2" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35149 } } peers { permanent_uuid: "bb3c93ab3a934d559efe4453b6da5868" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45519 } }
I20260620 20:27:11.621868 19495 raft_consensus.cc:374] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:11.621984 19495 raft_consensus.cc:729] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9d392805384d32b2c9ccff9eb973a1, State: Initialized, Role: FOLLOWER
I20260620 20:27:11.622023 19494 raft_consensus.cc:729] T 3d10d7bf748c4c84a42070b777e1eed1 P bb3c93ab3a934d559efe4453b6da5868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb3c93ab3a934d559efe4453b6da5868, State: Initialized, Role: FOLLOWER
I20260620 20:27:11.622609 19476 heartbeater.cc:503] Master 127.31.235.190:38661 was elected leader, sending a full tablet report...
I20260620 20:27:11.622709 19494 consensus_queue.cc:260] T 3d10d7bf748c4c84a42070b777e1eed1 P bb3c93ab3a934d559efe4453b6da5868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9d392805384d32b2c9ccff9eb973a1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36627 } } peers { permanent_uuid: "2ccf072073144251a1fa67e10bb3a3b2" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35149 } } peers { permanent_uuid: "bb3c93ab3a934d559efe4453b6da5868" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45519 } }
I20260620 20:27:11.623274 19496 ts_tablet_manager.cc:1428] T 3d10d7bf748c4c84a42070b777e1eed1 P 2ccf072073144251a1fa67e10bb3a3b2: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260620 20:27:11.623101 19495 consensus_queue.cc:260] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9d392805384d32b2c9ccff9eb973a1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36627 } } peers { permanent_uuid: "2ccf072073144251a1fa67e10bb3a3b2" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35149 } } peers { permanent_uuid: "bb3c93ab3a934d559efe4453b6da5868" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45519 } }
I20260620 20:27:11.624897 19494 ts_tablet_manager.cc:1428] T 3d10d7bf748c4c84a42070b777e1eed1 P bb3c93ab3a934d559efe4453b6da5868: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260620 20:27:11.626394 19346 heartbeater.cc:503] Master 127.31.235.190:38661 was elected leader, sending a full tablet report...
I20260620 20:27:11.626650 19495 ts_tablet_manager.cc:1428] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260620 20:27:11.657986 19502 raft_consensus.cc:482] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:27:11.658308 19502 raft_consensus.cc:504] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9d392805384d32b2c9ccff9eb973a1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36627 } } peers { permanent_uuid: "2ccf072073144251a1fa67e10bb3a3b2" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35149 } } peers { permanent_uuid: "bb3c93ab3a934d559efe4453b6da5868" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45519 } }
I20260620 20:27:11.660135 19502 leader_election.cc:288] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ccf072073144251a1fa67e10bb3a3b2 (127.31.235.131:35149), bb3c93ab3a934d559efe4453b6da5868 (127.31.235.129:45519)
I20260620 20:27:11.666823 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:11.660663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:35677 (local address 127.31.235.129:45519)
0620 20:27:11.661051 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:11.661060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:11.661724 (+ 664us) server_negotiation.cc:407] Connection header received
0620 20:27:11.661923 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:11.661928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:11.661997 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:11.662096 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:11.662631 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.663293 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.665050 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.665882 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.666067 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:11.666137 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:11.666428 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 20:27:11.666568 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260620 20:27:11.666832 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:11.660555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:34499 (local address 127.31.235.131:35149)
0620 20:27:11.660819 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:11.660839 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:11.661394 (+ 555us) server_negotiation.cc:407] Connection header received
0620 20:27:11.661692 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:11.661696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:11.661757 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:11.661855 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:11.662413 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.663141 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.665069 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.665909 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.666085 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:11.666151 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:11.666438 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 20:27:11.666568 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 20:27:11.666906 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:11.660350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:35149 (local address 127.31.235.130:34499)
0620 20:27:11.661275 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:11.661291 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:11.661477 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:11.661883 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:11.661888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:11.661912 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:11.662255 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:11.662272 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.663275 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.663281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:11.664874 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:11.664885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.666095 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.666101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:11.666287 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:11.666308 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 20:27:11.666412 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 20:27:11.666592 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":117,"thread_start_us":915,"threads_started":1}
I20260620 20:27:11.666962 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:11.660414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:45519 (local address 127.31.235.130:35677)
0620 20:27:11.661609 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:11.661624 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:11.661767 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:11.662111 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:11.662116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:11.662143 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:11.662489 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:11.662498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.663443 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.663449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:11.664860 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:11.664872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.666086 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.666092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:11.666260 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:11.666286 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 20:27:11.666390 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 20:27:11.666584 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"mutex_wait_us":46,"thread_start_us":482,"threads_started":1}
I20260620 20:27:11.668063 19431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d10d7bf748c4c84a42070b777e1eed1" candidate_uuid: "8a9d392805384d32b2c9ccff9eb973a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ccf072073144251a1fa67e10bb3a3b2" is_pre_election: true
I20260620 20:27:11.668197 19170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d10d7bf748c4c84a42070b777e1eed1" candidate_uuid: "8a9d392805384d32b2c9ccff9eb973a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb3c93ab3a934d559efe4453b6da5868" is_pre_election: true
I20260620 20:27:11.668706 19170 raft_consensus.cc:2454] T 3d10d7bf748c4c84a42070b777e1eed1 P bb3c93ab3a934d559efe4453b6da5868 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9d392805384d32b2c9ccff9eb973a1 in term 0.
I20260620 20:27:11.668678 19431 raft_consensus.cc:2454] T 3d10d7bf748c4c84a42070b777e1eed1 P 2ccf072073144251a1fa67e10bb3a3b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a9d392805384d32b2c9ccff9eb973a1 in term 0.
I20260620 20:27:11.669559 19236 leader_election.cc:302] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ccf072073144251a1fa67e10bb3a3b2, 8a9d392805384d32b2c9ccff9eb973a1; no voters:
I20260620 20:27:11.670001 19502 raft_consensus.cc:2781] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:27:11.670159 19502 raft_consensus.cc:482] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:27:11.670269 19502 raft_consensus.cc:3037] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:11.672487 19502 raft_consensus.cc:504] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9d392805384d32b2c9ccff9eb973a1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36627 } } peers { permanent_uuid: "2ccf072073144251a1fa67e10bb3a3b2" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35149 } } peers { permanent_uuid: "bb3c93ab3a934d559efe4453b6da5868" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45519 } }
I20260620 20:27:11.673055 19502 leader_election.cc:288] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 [CANDIDATE]: Term 1 election: Requested vote from peers 2ccf072073144251a1fa67e10bb3a3b2 (127.31.235.131:35149), bb3c93ab3a934d559efe4453b6da5868 (127.31.235.129:45519)
I20260620 20:27:11.673593 19170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d10d7bf748c4c84a42070b777e1eed1" candidate_uuid: "8a9d392805384d32b2c9ccff9eb973a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb3c93ab3a934d559efe4453b6da5868"
I20260620 20:27:11.673606 19431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d10d7bf748c4c84a42070b777e1eed1" candidate_uuid: "8a9d392805384d32b2c9ccff9eb973a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ccf072073144251a1fa67e10bb3a3b2"
I20260620 20:27:11.673804 19170 raft_consensus.cc:3037] T 3d10d7bf748c4c84a42070b777e1eed1 P bb3c93ab3a934d559efe4453b6da5868 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:11.673800 19431 raft_consensus.cc:3037] T 3d10d7bf748c4c84a42070b777e1eed1 P 2ccf072073144251a1fa67e10bb3a3b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:11.676008 19170 raft_consensus.cc:2454] T 3d10d7bf748c4c84a42070b777e1eed1 P bb3c93ab3a934d559efe4453b6da5868 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9d392805384d32b2c9ccff9eb973a1 in term 1.
I20260620 20:27:11.676030 19431 raft_consensus.cc:2454] T 3d10d7bf748c4c84a42070b777e1eed1 P 2ccf072073144251a1fa67e10bb3a3b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a9d392805384d32b2c9ccff9eb973a1 in term 1.
I20260620 20:27:11.676450 19235 leader_election.cc:302] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a9d392805384d32b2c9ccff9eb973a1, bb3c93ab3a934d559efe4453b6da5868; no voters:
I20260620 20:27:11.676770 19502 raft_consensus.cc:2781] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:11.677194 19502 raft_consensus.cc:686] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 [term 1 LEADER]: Becoming Leader. State: Replica: 8a9d392805384d32b2c9ccff9eb973a1, State: Running, Role: LEADER
I20260620 20:27:11.677724 19502 consensus_queue.cc:237] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9d392805384d32b2c9ccff9eb973a1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36627 } } peers { permanent_uuid: "2ccf072073144251a1fa67e10bb3a3b2" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35149 } } peers { permanent_uuid: "bb3c93ab3a934d559efe4453b6da5868" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45519 } }
I20260620 20:27:11.682391 19028 catalog_manager.cc:5495] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a9d392805384d32b2c9ccff9eb973a1 (127.31.235.130). New cstate: current_term: 1 leader_uuid: "8a9d392805384d32b2c9ccff9eb973a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a9d392805384d32b2c9ccff9eb973a1" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 36627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ccf072073144251a1fa67e10bb3a3b2" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb3c93ab3a934d559efe4453b6da5868" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45519 } health_report { overall_health: UNKNOWN } } }
I20260620 20:27:11.724998 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:11.719564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52562 (local address 127.31.235.130:36627)
0620 20:27:11.719780 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:11.719785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:11.719806 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:27:11.719997 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:11.720001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:11.720088 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:11.720240 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:27:11.720630 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.721357 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.722803 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.723201 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.723349 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:11.723928 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:27:11.724411 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:27:11.724602 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:11.724712 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 20:27:11.724755 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:27:11.733135 19170 raft_consensus.cc:1261] T 3d10d7bf748c4c84a42070b777e1eed1 P bb3c93ab3a934d559efe4453b6da5868 [term 1 FOLLOWER]: Refusing update from remote peer 8a9d392805384d32b2c9ccff9eb973a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:27:11.733134 19431 raft_consensus.cc:1261] T 3d10d7bf748c4c84a42070b777e1eed1 P 2ccf072073144251a1fa67e10bb3a3b2 [term 1 FOLLOWER]: Refusing update from remote peer 8a9d392805384d32b2c9ccff9eb973a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:27:11.734215 19505 consensus_queue.cc:1035] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ccf072073144251a1fa67e10bb3a3b2" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:27:11.734539 19502 consensus_queue.cc:1035] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb3c93ab3a934d559efe4453b6da5868" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 45519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:27:11.765786 19028 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:27:11.766091 19028 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:27:11.769591 19028 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 157780d97ac846af9352704822a81d21: Sending DeleteTablet for 3 replicas of tablet 3d10d7bf748c4c84a42070b777e1eed1
I20260620 20:27:11.770797 19411 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d10d7bf748c4c84a42070b777e1eed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:27:11 UTC) from {username='slave'} at 127.0.0.1:47244
I20260620 20:27:11.770792 19281 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d10d7bf748c4c84a42070b777e1eed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:27:11 UTC) from {username='slave'} at 127.0.0.1:52554
I20260620 20:27:11.771723 19520 tablet_replica.cc:330] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1: stopping tablet replica
I20260620 20:27:11.771718 19150 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d10d7bf748c4c84a42070b777e1eed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:27:11 UTC) from {username='slave'} at 127.0.0.1:53180
I20260620 20:27:11.772337 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 19088
I20260620 20:27:11.772661 19520 raft_consensus.cc:2229] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:27:11.773545 19520 raft_consensus.cc:2258] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:27:11.774410 19519 tablet_replica.cc:330] T 3d10d7bf748c4c84a42070b777e1eed1 P 2ccf072073144251a1fa67e10bb3a3b2: stopping tablet replica
I20260620 20:27:11.774641 19521 tablet_replica.cc:330] T 3d10d7bf748c4c84a42070b777e1eed1 P bb3c93ab3a934d559efe4453b6da5868: stopping tablet replica
I20260620 20:27:11.774806 19519 raft_consensus.cc:2229] T 3d10d7bf748c4c84a42070b777e1eed1 P 2ccf072073144251a1fa67e10bb3a3b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:27:11.775022 19521 raft_consensus.cc:2229] T 3d10d7bf748c4c84a42070b777e1eed1 P bb3c93ab3a934d559efe4453b6da5868 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:27:11.775345 19519 raft_consensus.cc:2258] T 3d10d7bf748c4c84a42070b777e1eed1 P 2ccf072073144251a1fa67e10bb3a3b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:27:11.775501 19521 raft_consensus.cc:2258] T 3d10d7bf748c4c84a42070b777e1eed1 P bb3c93ab3a934d559efe4453b6da5868 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:27:11.777678 19519 ts_tablet_manager.cc:1905] T 3d10d7bf748c4c84a42070b777e1eed1 P 2ccf072073144251a1fa67e10bb3a3b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:27:11.777685 19521 ts_tablet_manager.cc:1905] T 3d10d7bf748c4c84a42070b777e1eed1 P bb3c93ab3a934d559efe4453b6da5868: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:27:11.785025 19521 ts_tablet_manager.cc:1918] T 3d10d7bf748c4c84a42070b777e1eed1 P bb3c93ab3a934d559efe4453b6da5868: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:27:11.785111 19519 ts_tablet_manager.cc:1918] T 3d10d7bf748c4c84a42070b777e1eed1 P 2ccf072073144251a1fa67e10bb3a3b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:27:11.785233 19521 log.cc:1196] T 3d10d7bf748c4c84a42070b777e1eed1 P bb3c93ab3a934d559efe4453b6da5868: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/3d10d7bf748c4c84a42070b777e1eed1
I20260620 20:27:11.785310 19519 log.cc:1196] T 3d10d7bf748c4c84a42070b777e1eed1 P 2ccf072073144251a1fa67e10bb3a3b2: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/3d10d7bf748c4c84a42070b777e1eed1
I20260620 20:27:11.785717 19520 ts_tablet_manager.cc:1905] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:27:11.785861 19521 ts_tablet_manager.cc:1939] T 3d10d7bf748c4c84a42070b777e1eed1 P bb3c93ab3a934d559efe4453b6da5868: Deleting consensus metadata
I20260620 20:27:11.785900 19519 ts_tablet_manager.cc:1939] T 3d10d7bf748c4c84a42070b777e1eed1 P 2ccf072073144251a1fa67e10bb3a3b2: Deleting consensus metadata
I20260620 20:27:11.787752 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:11.774545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.31.235.129:45519)
0620 20:27:11.774726 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:11.774733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:11.777152 (+ 2419us) server_negotiation.cc:407] Connection header received
0620 20:27:11.777428 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:11.777434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:11.777553 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:11.777629 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:11.778263 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.779416 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.782638 (+ 3222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.783378 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.783489 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:11.786049 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:11.786062 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:11.786081 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:11.786857 (+ 776us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:11.786923 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:11.786946 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:11.787029 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:11.787273 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 20:27:11.787385 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 20:27:11.787767 19014 catalog_manager.cc:4841] TS bb3c93ab3a934d559efe4453b6da5868 (127.31.235.129:45519): tablet 3d10d7bf748c4c84a42070b777e1eed1 (table test-table [id=171c3f90503e43058a3bdde9bb490b6d]) successfully deleted
I20260620 20:27:11.787799 19015 catalog_manager.cc:4841] TS 2ccf072073144251a1fa67e10bb3a3b2 (127.31.235.131:35149): tablet 3d10d7bf748c4c84a42070b777e1eed1 (table test-table [id=171c3f90503e43058a3bdde9bb490b6d]) successfully deleted
I20260620 20:27:11.788887 19210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:11.796962 19520 ts_tablet_manager.cc:1918] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:27:11.797174 19520 log.cc:1196] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/3d10d7bf748c4c84a42070b777e1eed1
I20260620 20:27:11.797744 19520 ts_tablet_manager.cc:1939] T 3d10d7bf748c4c84a42070b777e1eed1 P 8a9d392805384d32b2c9ccff9eb973a1: Deleting consensus metadata
I20260620 20:27:11.799227 19014 catalog_manager.cc:4841] TS 8a9d392805384d32b2c9ccff9eb973a1 (127.31.235.130:36627): tablet 3d10d7bf748c4c84a42070b777e1eed1 (table test-table [id=171c3f90503e43058a3bdde9bb490b6d]) successfully deleted
I20260620 20:27:11.899932 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 19088
I20260620 20:27:11.917179 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 19219
I20260620 20:27:11.924741 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:11.918040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52568 (local address 127.31.235.130:36627)
0620 20:27:11.918223 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:11.918229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:11.918249 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:27:11.918457 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:11.918462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:11.918554 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:11.918673 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:11.919087 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.919853 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.921164 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:11.921824 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:11.921958 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:11.923084 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:11.923093 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:11.923112 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:11.923831 (+ 719us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:11.923883 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:11.923905 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:11.924007 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:11.924197 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 20:27:11.924331 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:27:11.925469 19341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:12.037393 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 19219
I20260620 20:27:12.056203 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 19349
I20260620 20:27:12.062816 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:12.056965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.31.235.131:35149)
0620 20:27:12.057128 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:12.057132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:12.057153 (+ 21us) server_negotiation.cc:407] Connection header received
0620 20:27:12.057333 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:12.057337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:12.057428 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:12.057567 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:12.057954 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:12.058743 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:12.060069 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:12.060451 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:12.060546 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:12.061616 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:12.061625 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:12.061639 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:12.062176 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:12.062221 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:12.062245 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:12.062368 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:12.062497 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 20:27:12.062581 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 20:27:12.063474 19471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:12.152191 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 19349
I20260620 20:27:12.169039 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 18998
I20260620 20:27:12.170306 19059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:12.283711 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 18998
2026-06-20T20:27:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2810 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:27:12 dist-test-slave-zzwf krb5kdc[19529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:27:12 dist-test-slave-zzwf krb5kdc[19529](info): set up 2 sockets
Jun 20 20:27:12 dist-test-slave-zzwf krb5kdc[19529](info): commencing operation
krb5kdc: starting...
W20260620 20:27:13.406070 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.009s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:27:13 dist-test-slave-zzwf krb5kdc[19529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:27:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:27:13.665827 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:46295
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:46295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:13.995146 19542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:13.995460 19542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:13.995667 19542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:14.002919 19542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:14.003019 19542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:14.003120 19542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:14.003208 19542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:14.003322 19542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:14.013129 19542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:46295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:46295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:14.014453 19542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:14.015776 19542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:14.027609 19550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:14.027652 19548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:14.028187 19547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:14.028461 19542 server_base.cc:1034] running on GCE node
I20260620 20:27:14.029124 19542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:14.030526 19542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:14.031733 19542 hybrid_clock.cc:648] HybridClock initialized: now 1781987234031692 us; error 33 us; skew 500 ppm
I20260620 20:27:14.032025 19542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:14 dist-test-slave-zzwf krb5kdc[19529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987234, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:14.040145 19542 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:27:14.041612 19542 webserver.cc:460] Webserver started at http://127.31.235.190:46617/ using document root <none> and password file <none>
I20260620 20:27:14.042253 19542 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:14.042351 19542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:14.042585 19542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:14.045403 19542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "b46a1646e6614cb584877d216f30dbb0"
format_stamp: "Formatted at 2026-06-20 20:27:14 on dist-test-slave-zzwf"
server_key: "7006e7f1adce5cbe7db8e415b16a5ef4"
server_key_iv: "1adbbfe34b0175518e82b54938bd7638"
server_key_version: "encryptionkey@0"
I20260620 20:27:14.046129 19542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "b46a1646e6614cb584877d216f30dbb0"
format_stamp: "Formatted at 2026-06-20 20:27:14 on dist-test-slave-zzwf"
server_key: "7006e7f1adce5cbe7db8e415b16a5ef4"
server_key_iv: "1adbbfe34b0175518e82b54938bd7638"
server_key_version: "encryptionkey@0"
I20260620 20:27:14.050948 19542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 20:27:14.054167 19557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:14.055555 19542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:27:14.055696 19542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "b46a1646e6614cb584877d216f30dbb0"
format_stamp: "Formatted at 2026-06-20 20:27:14 on dist-test-slave-zzwf"
server_key: "7006e7f1adce5cbe7db8e415b16a5ef4"
server_key_iv: "1adbbfe34b0175518e82b54938bd7638"
server_key_version: "encryptionkey@0"
I20260620 20:27:14.055861 19542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:14.088294 19542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:14.092726 19542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:14.093009 19542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:14.112798 19542 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:46295
I20260620 20:27:14.112808 19608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:46295 every 8 connection(s)
I20260620 20:27:14.114300 19542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:14.118033 19609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:14.120726 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 19542
I20260620 20:27:14.120919 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:14.121414 32686 external_mini_cluster.cc:1428] Setting key 5a2ccddb87e476945792ce3f9b4074de
I20260620 20:27:14.130380 19609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0: Bootstrap starting.
I20260620 20:27:14.134207 19609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:14.135563 19609 log.cc:824] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:27:14 dist-test-slave-zzwf krb5kdc[19529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:27:14.138638 19609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0: No bootstrap required, opened a new log
I20260620 20:27:14.145763 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:14.123470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50926 (local address 127.31.235.190:46295)
0620 20:27:14.124332 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:14.124344 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:14.124387 (+ 43us) server_negotiation.cc:407] Connection header received
0620 20:27:14.126258 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:14.126282 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:14.126725 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:14.127107 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:14.128958 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:14.130464 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:14.132302 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:14.133032 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:14.133170 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:14.138216 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:14.138224 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:14.138246 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:14.138280 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:14.141478 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:14.142297 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:14.142306 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:14.142317 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:14.142478 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:14.142911 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:14.142921 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:14.142923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:14.143458 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:14.143795 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:14.144277 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 20:27:14.144683 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":420,"threads_started":1}
I20260620 20:27:14.146524 19609 raft_consensus.cc:348] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b46a1646e6614cb584877d216f30dbb0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46295 } }
I20260620 20:27:14.146893 19609 raft_consensus.cc:374] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:14.147008 19609 raft_consensus.cc:729] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b46a1646e6614cb584877d216f30dbb0, State: Initialized, Role: FOLLOWER
I20260620 20:27:14.147528 19609 consensus_queue.cc:260] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b46a1646e6614cb584877d216f30dbb0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46295 } }
I20260620 20:27:14.147778 19609 raft_consensus.cc:388] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:14.147934 19609 raft_consensus.cc:482] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:14.148114 19609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:14.150432 19609 raft_consensus.cc:504] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b46a1646e6614cb584877d216f30dbb0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46295 } }
I20260620 20:27:14.150906 19609 leader_election.cc:302] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b46a1646e6614cb584877d216f30dbb0; no voters:
I20260620 20:27:14.151456 19609 leader_election.cc:288] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:14.151605 19614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:14.152379 19614 raft_consensus.cc:686] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0 [term 1 LEADER]: Becoming Leader. State: Replica: b46a1646e6614cb584877d216f30dbb0, State: Running, Role: LEADER
I20260620 20:27:14.152907 19614 consensus_queue.cc:237] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b46a1646e6614cb584877d216f30dbb0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46295 } }
I20260620 20:27:14.153621 19609 sys_catalog.cc:564] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:14.156407 19616 sys_catalog.cc:455] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b46a1646e6614cb584877d216f30dbb0. Latest consensus state: current_term: 1 leader_uuid: "b46a1646e6614cb584877d216f30dbb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b46a1646e6614cb584877d216f30dbb0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46295 } } }
I20260620 20:27:14.156581 19615 sys_catalog.cc:455] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b46a1646e6614cb584877d216f30dbb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b46a1646e6614cb584877d216f30dbb0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46295 } } }
I20260620 20:27:14.156867 19616 sys_catalog.cc:458] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:14.156968 19615 sys_catalog.cc:458] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:14.159446 19622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:14.164114 19622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:14.172302 19622 catalog_manager.cc:1347] Generated new cluster ID: 7f8e649f0de5446f84421fdab79e546d
I20260620 20:27:14.172415 19622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:14.184762 19622 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:14.186143 19622 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:14.194154 19622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0: Generated new TSK 0
I20260620 20:27:14.194952 19622 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.235.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
Entry for principal kudu/127.31.235.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab.
I20260620 20:27:14.261479 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:46295
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:14.591938 19635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:14.592221 19635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:14.592315 19635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:14.599882 19635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:14.599997 19635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:14.600113 19635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:27:14.609983 19635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.129.keytab
--principal=kudu/127.31.235.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:46295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:14.611271 19635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:14.612517 19635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:14.626309 19641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:14.626305 19643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:14.626607 19640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:14.627234 19635 server_base.cc:1034] running on GCE node
I20260620 20:27:14.627878 19635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:14.628859 19635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:14.630085 19635 hybrid_clock.cc:648] HybridClock initialized: now 1781987234630026 us; error 53 us; skew 500 ppm
I20260620 20:27:14.630409 19635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:14 dist-test-slave-zzwf krb5kdc[19529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987234, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:14.633410 19635 init.cc:377] Logged in from keytab as kudu/127.31.235.129@KRBTEST.COM (short username kudu)
I20260620 20:27:14.634963 19635 webserver.cc:460] Webserver started at http://127.31.235.129:46113/ using document root <none> and password file <none>
I20260620 20:27:14.635644 19635 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:14.635784 19635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:14.636098 19635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:14.639004 19635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "0c35137799034ebab3db71ae171a48db"
format_stamp: "Formatted at 2026-06-20 20:27:14 on dist-test-slave-zzwf"
server_key: "42459eb2faa1aa8710d96016d6e90610"
server_key_iv: "072bc6e05873c527f42288c2222452f8"
server_key_version: "encryptionkey@0"
I20260620 20:27:14.639751 19635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "0c35137799034ebab3db71ae171a48db"
format_stamp: "Formatted at 2026-06-20 20:27:14 on dist-test-slave-zzwf"
server_key: "42459eb2faa1aa8710d96016d6e90610"
server_key_iv: "072bc6e05873c527f42288c2222452f8"
server_key_version: "encryptionkey@0"
I20260620 20:27:14.644891 19635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 20:27:14.648411 19650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:14.649791 19635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:27:14.649977 19635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "0c35137799034ebab3db71ae171a48db"
format_stamp: "Formatted at 2026-06-20 20:27:14 on dist-test-slave-zzwf"
server_key: "42459eb2faa1aa8710d96016d6e90610"
server_key_iv: "072bc6e05873c527f42288c2222452f8"
server_key_version: "encryptionkey@0"
I20260620 20:27:14.650182 19635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:14.669752 19635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:14.673810 19635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:14.674129 19635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:14.675112 19635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:27:14.676484 19635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:27:14.676592 19635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:14.676751 19635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:27:14.676827 19635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:14.711901 19635 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:37153
I20260620 20:27:14.711987 19762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:37153 every 8 connection(s)
I20260620 20:27:14.713171 19635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:27:14.716506 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 19635
I20260620 20:27:14.716679 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:27:14.717245 32686 external_mini_cluster.cc:1428] Setting key 686fb498d08b80ad3af34a3cfcc32c3a
Jun 20 20:27:14 dist-test-slave-zzwf krb5kdc[19529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987234, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.129@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:27:14.737044 19765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:14.715754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:42167 (local address 127.31.235.190:46295)
0620 20:27:14.716249 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:14.716258 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:14.717408 (+ 1150us) server_negotiation.cc:407] Connection header received
0620 20:27:14.719359 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:14.719364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:14.719425 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:14.719513 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:14.721707 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:14.722408 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:14.724866 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:14.725325 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:14.725451 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:14.729990 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:14.729999 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:14.730007 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:14.730052 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:14.733628 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:14.734415 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:14.734423 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:14.734429 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:14.734580 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:14.735018 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:14.735025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:14.735031 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:14.735405 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:14.735556 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:14.736279 (+ 723us) server_negotiation.cc:299] Negotiation successful
0620 20:27:14.736551 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":209,"threads_started":1}
I20260620 20:27:14.737706 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:14.716295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46295 (local address 127.31.235.129:42167)
0620 20:27:14.717203 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:14.717279 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:14.719061 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:14.719797 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:14.719809 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:14.720634 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:14.721522 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:14.721551 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:14.722569 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:14.722577 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:14.724697 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:14.724714 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:14.725614 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:14.725624 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:14.725952 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:14.726764 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:27:14.726804 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:14.729738 (+ 2934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:14.733793 (+ 4055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:14.733801 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:14.733816 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:14.734273 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:14.734689 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:14.734692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:14.734695 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:14.734881 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:14.735519 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:14.735525 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:14.735871 (+ 346us) client_negotiation.cc:769] Sending connection context
0620 20:27:14.736117 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 20:27:14.736514 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":345,"threads_started":1}
I20260620 20:27:14.739382 19763 heartbeater.cc:347] Connected to a master server at 127.31.235.190:46295
I20260620 20:27:14.739691 19763 heartbeater.cc:464] Registering TS with master...
I20260620 20:27:14.740439 19763 heartbeater.cc:511] Master 127.31.235.190:46295 requested a full tablet report, sending...
I20260620 20:27:14.742906 19574 ts_manager.cc:194] Registered new tserver with Master: 0c35137799034ebab3db71ae171a48db (127.31.235.129:37153)
I20260620 20:27:14.745095 19574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.129@KRBTEST.COM'} at 127.31.235.129:42167
WARNING: no policy specified for kudu/127.31.235.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
Entry for principal kudu/127.31.235.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab.
I20260620 20:27:14.780611 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:46295
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:15.117081 19769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:15.117439 19769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:15.117610 19769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:15.130597 19769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:15.130762 19769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:15.130967 19769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:27:15.144090 19769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.130.keytab
--principal=kudu/127.31.235.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:46295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:15.145328 19769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:15.146643 19769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:15.160629 19774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:15.160857 19775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:15.162102 19777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:15.162559 19769 server_base.cc:1034] running on GCE node
I20260620 20:27:15.163215 19769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:15.164265 19769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:15.165577 19769 hybrid_clock.cc:648] HybridClock initialized: now 1781987235165439 us; error 71 us; skew 500 ppm
I20260620 20:27:15.165942 19769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:15 dist-test-slave-zzwf krb5kdc[19529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987235, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:15.169193 19769 init.cc:377] Logged in from keytab as kudu/127.31.235.130@KRBTEST.COM (short username kudu)
I20260620 20:27:15.170787 19769 webserver.cc:460] Webserver started at http://127.31.235.130:36197/ using document root <none> and password file <none>
I20260620 20:27:15.171474 19769 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:15.171576 19769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:15.171829 19769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:15.175671 19769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "77af88fddd76447ead6eba1718e8755e"
format_stamp: "Formatted at 2026-06-20 20:27:15 on dist-test-slave-zzwf"
server_key: "6a39238c92e65e82f05b0aafe6e6285e"
server_key_iv: "da011eb2e842e90c5439a003e3d9e764"
server_key_version: "encryptionkey@0"
I20260620 20:27:15.176452 19769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "77af88fddd76447ead6eba1718e8755e"
format_stamp: "Formatted at 2026-06-20 20:27:15 on dist-test-slave-zzwf"
server_key: "6a39238c92e65e82f05b0aafe6e6285e"
server_key_iv: "da011eb2e842e90c5439a003e3d9e764"
server_key_version: "encryptionkey@0"
I20260620 20:27:15.182788 19769 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 20:27:15.186614 19784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:15.188238 19769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:27:15.188409 19769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "77af88fddd76447ead6eba1718e8755e"
format_stamp: "Formatted at 2026-06-20 20:27:15 on dist-test-slave-zzwf"
server_key: "6a39238c92e65e82f05b0aafe6e6285e"
server_key_iv: "da011eb2e842e90c5439a003e3d9e764"
server_key_version: "encryptionkey@0"
I20260620 20:27:15.188606 19769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:15.208734 19769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:15.213173 19769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:15.213423 19769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:15.214442 19769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:27:15.215916 19769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:27:15.215991 19769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:15.216125 19769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:27:15.216176 19769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:15.252449 19769 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:34915
I20260620 20:27:15.252523 19896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:34915 every 8 connection(s)
I20260620 20:27:15.254263 19769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:27:15.256562 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 19769
I20260620 20:27:15.256743 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:27:15.257361 32686 external_mini_cluster.cc:1428] Setting key 401309a6b8cc74a8da712085cccc0274
Jun 20 20:27:15 dist-test-slave-zzwf krb5kdc[19529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987235, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.130@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:27:15.281855 19899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:15.257236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:49301 (local address 127.31.235.190:46295)
0620 20:27:15.257744 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:15.257750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:15.258844 (+ 1094us) server_negotiation.cc:407] Connection header received
0620 20:27:15.262024 (+ 3180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:15.262033 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:15.262125 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:15.262244 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:15.264825 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.266000 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.268595 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.269293 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.269437 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:15.273798 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:15.273803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:15.273826 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:15.273874 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:15.277899 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:15.278990 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:15.278997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:15.279000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:15.279171 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:15.279743 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:15.279749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:15.279754 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:15.280169 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:15.280334 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:15.281175 (+ 841us) server_negotiation.cc:299] Negotiation successful
0620 20:27:15.281416 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":318,"threads_started":1}
I20260620 20:27:15.283061 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:15.257806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46295 (local address 127.31.235.130:49301)
0620 20:27:15.258627 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:15.258686 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:15.261604 (+ 2918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:15.262509 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:15.262522 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:15.263523 (+ 1001us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:15.264545 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:15.264578 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.266164 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.266169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:15.268433 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:15.268451 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.269558 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.269568 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:15.269906 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:15.270790 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:27:15.270838 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:15.273467 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:15.278100 (+ 4633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:15.278113 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:15.278135 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:15.278800 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:15.279321 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:15.279329 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:15.279336 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:15.279598 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:15.280330 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:15.280340 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:15.280718 (+ 378us) client_negotiation.cc:769] Sending connection context
0620 20:27:15.281094 (+ 376us) client_negotiation.cc:240] Negotiation successful
0620 20:27:15.281731 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":318,"threads_started":1}
I20260620 20:27:15.284795 19897 heartbeater.cc:347] Connected to a master server at 127.31.235.190:46295
I20260620 20:27:15.285158 19897 heartbeater.cc:464] Registering TS with master...
I20260620 20:27:15.286164 19897 heartbeater.cc:511] Master 127.31.235.190:46295 requested a full tablet report, sending...
I20260620 20:27:15.288056 19574 ts_manager.cc:194] Registered new tserver with Master: 77af88fddd76447ead6eba1718e8755e (127.31.235.130:34915)
WARNING: no policy specified for kudu/127.31.235.131@KRBTEST.COM; defaulting to no policy
I20260620 20:27:15.288959 19574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.130@KRBTEST.COM'} at 127.31.235.130:49301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
Entry for principal kudu/127.31.235.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab.
I20260620 20:27:15.320791 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:46295
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:15.656502 19903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:15.656857 19903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:15.657037 19903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:15.664970 19903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:15.665153 19903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:15.665355 19903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:27:15.675701 19903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.131.keytab
--principal=kudu/127.31.235.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:46295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:15.677160 19903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:15.678576 19903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:15.691968 19908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:15.691926 19909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:15.692740 19911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:15.693965 19903 server_base.cc:1034] running on GCE node
I20260620 20:27:15.694620 19903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:15.695652 19903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:15.696906 19903 hybrid_clock.cc:648] HybridClock initialized: now 1781987235696860 us; error 39 us; skew 500 ppm
I20260620 20:27:15.697247 19903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:15 dist-test-slave-zzwf krb5kdc[19529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987235, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:15.700503 19903 init.cc:377] Logged in from keytab as kudu/127.31.235.131@KRBTEST.COM (short username kudu)
I20260620 20:27:15.702159 19903 webserver.cc:460] Webserver started at http://127.31.235.131:44771/ using document root <none> and password file <none>
I20260620 20:27:15.702898 19903 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:15.703044 19903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:15.703365 19903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:15.706398 19903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "ed36b3dd6b804e14ac55e671b21dc84b"
format_stamp: "Formatted at 2026-06-20 20:27:15 on dist-test-slave-zzwf"
server_key: "5318ea7fe74f3a0a4d09dca5ab4bfb21"
server_key_iv: "be3f456349ee540565d8fb770774cfcb"
server_key_version: "encryptionkey@0"
I20260620 20:27:15.707180 19903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "ed36b3dd6b804e14ac55e671b21dc84b"
format_stamp: "Formatted at 2026-06-20 20:27:15 on dist-test-slave-zzwf"
server_key: "5318ea7fe74f3a0a4d09dca5ab4bfb21"
server_key_iv: "be3f456349ee540565d8fb770774cfcb"
server_key_version: "encryptionkey@0"
I20260620 20:27:15.712611 19903 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 20:27:15.716327 19918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:15.717662 19903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:27:15.717825 19903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "ed36b3dd6b804e14ac55e671b21dc84b"
format_stamp: "Formatted at 2026-06-20 20:27:15 on dist-test-slave-zzwf"
server_key: "5318ea7fe74f3a0a4d09dca5ab4bfb21"
server_key_iv: "be3f456349ee540565d8fb770774cfcb"
server_key_version: "encryptionkey@0"
I20260620 20:27:15.718060 19903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:15.737025 19903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:15.741391 19903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:15.741732 19903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:15.742710 19903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:27:15.744056 19903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:27:15.744161 19903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:15.744307 19903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:27:15.744414 19903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:15.748664 19763 heartbeater.cc:503] Master 127.31.235.190:46295 was elected leader, sending a full tablet report...
I20260620 20:27:15.779486 19903 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:35995
I20260620 20:27:15.779533 20030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:35995 every 8 connection(s)
I20260620 20:27:15.780766 19903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:27:15.785285 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 19903
I20260620 20:27:15.785463 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:27:15.786000 32686 external_mini_cluster.cc:1428] Setting key 7932c055cd6510206723f68f8161d10b
Jun 20 20:27:15 dist-test-slave-zzwf krb5kdc[19529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987235, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.131@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:27:15.800777 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:15.783400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:33705 (local address 127.31.235.190:46295)
0620 20:27:15.783898 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:15.783902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:15.784808 (+ 906us) server_negotiation.cc:407] Connection header received
0620 20:27:15.786798 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:15.786804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:15.786925 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:15.787048 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:15.788581 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.789364 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.790940 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.791340 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.791470 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:15.795187 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:15.795191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:15.795195 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:15.795228 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:15.797724 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:15.798466 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:15.798477 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:15.798479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:15.798595 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:15.799027 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:15.799032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:15.799035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:15.799303 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:15.799474 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:15.800161 (+ 687us) server_negotiation.cc:299] Negotiation successful
0620 20:27:15.800387 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":244,"threads_started":1}
I20260620 20:27:15.801823 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:15.783984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46295 (local address 127.31.235.131:33705)
0620 20:27:15.784647 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:15.784689 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:15.786525 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:15.787291 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:15.787300 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:15.787825 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:15.788414 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:15.788431 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.789539 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.789545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:15.790787 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:15.790799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.791579 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.791586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:15.791821 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:15.792562 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:27:15.792588 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:15.794942 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:15.797868 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:15.797879 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:15.797898 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:15.798323 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:15.798708 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:15.798713 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:15.798716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:15.798910 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:15.799432 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:15.799441 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:15.799790 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 20:27:15.800053 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 20:27:15.800604 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":251,"threads_started":1}
I20260620 20:27:15.803139 20031 heartbeater.cc:347] Connected to a master server at 127.31.235.190:46295
I20260620 20:27:15.803459 20031 heartbeater.cc:464] Registering TS with master...
I20260620 20:27:15.804170 20031 heartbeater.cc:511] Master 127.31.235.190:46295 requested a full tablet report, sending...
I20260620 20:27:15.805722 19574 ts_manager.cc:194] Registered new tserver with Master: ed36b3dd6b804e14ac55e671b21dc84b (127.31.235.131:35995)
I20260620 20:27:15.806610 19574 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.235.131@KRBTEST.COM'} at 127.31.235.131:33705
I20260620 20:27:15.813239 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:27:15.829865 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:15.817248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50932 (local address 127.31.235.190:46295)
0620 20:27:15.817461 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:15.817469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:15.817893 (+ 424us) server_negotiation.cc:407] Connection header received
0620 20:27:15.818023 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:15.818027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:15.818097 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:15.818175 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:15.819473 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.820173 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.821606 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.822074 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.822209 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:15.824489 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:15.824495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:15.824502 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:15.824538 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:15.827208 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:15.827869 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:15.827875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:15.827878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:15.827998 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:15.828384 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:15.828390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:15.828393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:15.828704 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:15.828962 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:15.829311 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 20:27:15.829549 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:27:15.833679 19574 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:27:15.840600 19574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:27:15.866286 20045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:15.852922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54418 (local address 127.31.235.130:34915)
0620 20:27:15.853646 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:15.853656 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:15.854552 (+ 896us) server_negotiation.cc:407] Connection header received
0620 20:27:15.854988 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:15.854995 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:15.855318 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:15.855496 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:15.856575 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.858503 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.862143 (+ 3640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.864025 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.864194 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:15.864375 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:15.865603 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0620 20:27:15.865860 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":946,"threads_started":1}
I20260620 20:27:15.867973 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:15.852736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:34915 (local address 127.0.0.1:54418)
0620 20:27:15.854340 (+ 1604us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:15.854388 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:15.854716 (+ 328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:15.855854 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:15.855865 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:15.855903 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:15.856350 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:15.856366 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.858710 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.858718 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:15.861934 (+ 3216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:15.861954 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.864559 (+ 2605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.864568 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:15.864875 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:15.864946 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 20:27:15.865112 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 20:27:15.865362 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1225,"thread_start_us":1084,"threads_started":1}
I20260620 20:27:15.871927 19832 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2443cba3e945ddb587c52b1f01bbad (DEFAULT_TABLE table=test-table [id=6f9cd175eeed475da6caf22d53e2d4ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:27:15.873328 19832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2443cba3e945ddb587c52b1f01bbad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:15.873679 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:15.853699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46776 (local address 127.31.235.129:37153)
0620 20:27:15.864532 (+ 10833us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:15.864545 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:15.864582 (+ 37us) server_negotiation.cc:407] Connection header received
0620 20:27:15.864702 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:15.864712 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:15.865028 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:15.865267 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:15.866966 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.868546 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.870862 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.872351 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.872548 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:15.872701 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:15.873151 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 20:27:15.873322 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10585,"thread_start_us":10380,"threads_started":1}
I20260620 20:27:15.873709 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:15.853321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:37153 (local address 127.0.0.1:46776)
0620 20:27:15.863431 (+ 10110us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:15.863454 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:15.863562 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:15.865654 (+ 2092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:15.865662 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:15.865697 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:15.866773 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:15.866787 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.868720 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.868729 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:15.870702 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:15.870714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.872674 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.872681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:15.872946 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:15.872981 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 20:27:15.873063 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 20:27:15.873192 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9982,"mutex_wait_us":39,"thread_start_us":9846,"threads_started":1}
I20260620 20:27:15.876063 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:15.854745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:35995 (local address 127.0.0.1:41990)
0620 20:27:15.867721 (+ 12976us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:15.867735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:15.867890 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:15.868529 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:15.868534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:15.868557 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:15.868985 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:15.868994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.870895 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.870900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:15.872915 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:15.872927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.875224 (+ 2297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.875230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:15.875432 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:15.875464 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 20:27:15.875562 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 20:27:15.875685 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12880,"mutex_wait_us":5,"thread_start_us":12746,"threads_started":1}
I20260620 20:27:15.876147 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:15.854986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41990 (local address 127.31.235.131:35995)
0620 20:27:15.867374 (+ 12388us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:15.867379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:15.867856 (+ 477us) server_negotiation.cc:407] Connection header received
0620 20:27:15.868092 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:15.868099 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:15.868331 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:15.868526 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:15.869163 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.870747 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.873064 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:15.875004 (+ 1940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:15.875224 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:15.875387 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:15.875623 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 20:27:15.875783 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12237,"thread_start_us":12102,"threads_started":1}
I20260620 20:27:15.877732 19698 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2443cba3e945ddb587c52b1f01bbad (DEFAULT_TABLE table=test-table [id=6f9cd175eeed475da6caf22d53e2d4ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:27:15.879241 19698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2443cba3e945ddb587c52b1f01bbad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:15.880151 19966 tablet_service.cc:1467] Processing CreateTablet for tablet 0d2443cba3e945ddb587c52b1f01bbad (DEFAULT_TABLE table=test-table [id=6f9cd175eeed475da6caf22d53e2d4ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:27:15.881551 19966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d2443cba3e945ddb587c52b1f01bbad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:15.886387 20050 tablet_bootstrap.cc:492] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e: Bootstrap starting.
I20260620 20:27:15.889405 20050 tablet_bootstrap.cc:654] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:15.890452 20050 log.cc:824] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:15.893630 20050 tablet_bootstrap.cc:492] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e: No bootstrap required, opened a new log
I20260620 20:27:15.893827 20052 tablet_bootstrap.cc:492] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db: Bootstrap starting.
I20260620 20:27:15.894037 20050 ts_tablet_manager.cc:1397] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 20:27:15.896023 20053 tablet_bootstrap.cc:492] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b: Bootstrap starting.
I20260620 20:27:15.898350 20052 tablet_bootstrap.cc:654] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:15.899175 20053 tablet_bootstrap.cc:654] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:15.899587 20052 log.cc:824] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:15.900205 20053 log.cc:824] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:15.903407 20052 tablet_bootstrap.cc:492] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db: No bootstrap required, opened a new log
I20260620 20:27:15.903678 20052 ts_tablet_manager.cc:1397] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260620 20:27:15.904711 20050 raft_consensus.cc:348] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77af88fddd76447ead6eba1718e8755e" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34915 } } peers { permanent_uuid: "ed36b3dd6b804e14ac55e671b21dc84b" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35995 } } peers { permanent_uuid: "0c35137799034ebab3db71ae171a48db" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37153 } }
I20260620 20:27:15.905169 20050 raft_consensus.cc:374] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:15.905279 20050 raft_consensus.cc:729] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77af88fddd76447ead6eba1718e8755e, State: Initialized, Role: FOLLOWER
I20260620 20:27:15.905936 20050 consensus_queue.cc:260] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77af88fddd76447ead6eba1718e8755e" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34915 } } peers { permanent_uuid: "ed36b3dd6b804e14ac55e671b21dc84b" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35995 } } peers { permanent_uuid: "0c35137799034ebab3db71ae171a48db" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37153 } }
I20260620 20:27:15.907151 19897 heartbeater.cc:503] Master 127.31.235.190:46295 was elected leader, sending a full tablet report...
I20260620 20:27:15.907779 20050 ts_tablet_manager.cc:1428] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260620 20:27:15.909266 20053 tablet_bootstrap.cc:492] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b: No bootstrap required, opened a new log
I20260620 20:27:15.909601 20053 ts_tablet_manager.cc:1397] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260620 20:27:15.913964 20052 raft_consensus.cc:348] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77af88fddd76447ead6eba1718e8755e" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34915 } } peers { permanent_uuid: "ed36b3dd6b804e14ac55e671b21dc84b" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35995 } } peers { permanent_uuid: "0c35137799034ebab3db71ae171a48db" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37153 } }
I20260620 20:27:15.914449 20052 raft_consensus.cc:374] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:15.914564 20052 raft_consensus.cc:729] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c35137799034ebab3db71ae171a48db, State: Initialized, Role: FOLLOWER
I20260620 20:27:15.915141 20052 consensus_queue.cc:260] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77af88fddd76447ead6eba1718e8755e" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34915 } } peers { permanent_uuid: "ed36b3dd6b804e14ac55e671b21dc84b" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35995 } } peers { permanent_uuid: "0c35137799034ebab3db71ae171a48db" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37153 } }
I20260620 20:27:15.916898 20052 ts_tablet_manager.cc:1428] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20260620 20:27:15.918699 20053 raft_consensus.cc:348] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77af88fddd76447ead6eba1718e8755e" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34915 } } peers { permanent_uuid: "ed36b3dd6b804e14ac55e671b21dc84b" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35995 } } peers { permanent_uuid: "0c35137799034ebab3db71ae171a48db" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37153 } }
I20260620 20:27:15.919082 20053 raft_consensus.cc:374] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:15.919189 20053 raft_consensus.cc:729] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed36b3dd6b804e14ac55e671b21dc84b, State: Initialized, Role: FOLLOWER
I20260620 20:27:15.919814 20053 consensus_queue.cc:260] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77af88fddd76447ead6eba1718e8755e" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34915 } } peers { permanent_uuid: "ed36b3dd6b804e14ac55e671b21dc84b" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35995 } } peers { permanent_uuid: "0c35137799034ebab3db71ae171a48db" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37153 } }
I20260620 20:27:15.920843 20031 heartbeater.cc:503] Master 127.31.235.190:46295 was elected leader, sending a full tablet report...
I20260620 20:27:15.921600 20053 ts_tablet_manager.cc:1428] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
W20260620 20:27:15.965579 19764 tablet.cc:2369] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:27:16.006548 19898 tablet.cc:2369] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 20:27:16.032457 20032 tablet.cc:2369] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:27:16.034621 20056 raft_consensus.cc:482] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:27:16.034894 20056 raft_consensus.cc:504] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77af88fddd76447ead6eba1718e8755e" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34915 } } peers { permanent_uuid: "ed36b3dd6b804e14ac55e671b21dc84b" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35995 } } peers { permanent_uuid: "0c35137799034ebab3db71ae171a48db" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37153 } }
I20260620 20:27:16.036521 20056 leader_election.cc:288] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed36b3dd6b804e14ac55e671b21dc84b (127.31.235.131:35995), 0c35137799034ebab3db71ae171a48db (127.31.235.129:37153)
I20260620 20:27:16.043725 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:16.036976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:50097 (local address 127.31.235.131:35995)
0620 20:27:16.037192 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:16.037198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:16.038236 (+ 1038us) server_negotiation.cc:407] Connection header received
0620 20:27:16.038434 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:16.038440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:16.038508 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:16.038606 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:16.039225 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:16.039943 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:16.041942 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:16.042687 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:16.042877 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:16.042959 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:16.043309 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 20:27:16.043447 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:27:16.043746 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:16.036976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:55607 (local address 127.31.235.129:37153)
0620 20:27:16.037192 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:16.037198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:16.038108 (+ 910us) server_negotiation.cc:407] Connection header received
0620 20:27:16.038338 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:16.038344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:16.038412 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:16.038513 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:16.039086 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:16.039791 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:16.041949 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:16.042731 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:16.042921 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:16.042998 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:16.043334 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 20:27:16.043470 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:27:16.044023 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:16.037056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:35995 (local address 127.31.235.130:50097)
0620 20:27:16.038117 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:16.038131 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:16.038296 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:16.038625 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:16.038629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:16.038667 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:16.039078 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:16.039087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:16.040093 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:16.040108 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:16.041793 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:16.041805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:16.042924 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:16.042932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:16.043130 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:16.043162 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 20:27:16.043274 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 20:27:16.043452 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":115,"thread_start_us":577,"threads_started":1}
I20260620 20:27:16.044334 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:16.037042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:37153 (local address 127.31.235.130:55607)
0620 20:27:16.037987 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:16.038005 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:16.038186 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:16.038530 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:16.038534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:16.038558 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:16.038928 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:16.038938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:16.039925 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:16.039931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:16.041792 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:16.041805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:16.042960 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:16.042966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:16.043144 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:16.043167 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 20:27:16.043310 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 20:27:16.043460 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"mutex_wait_us":60,"thread_start_us":513,"threads_started":1}
I20260620 20:27:16.045172 19986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2443cba3e945ddb587c52b1f01bbad" candidate_uuid: "77af88fddd76447ead6eba1718e8755e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed36b3dd6b804e14ac55e671b21dc84b" is_pre_election: true
I20260620 20:27:16.045318 19718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2443cba3e945ddb587c52b1f01bbad" candidate_uuid: "77af88fddd76447ead6eba1718e8755e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c35137799034ebab3db71ae171a48db" is_pre_election: true
I20260620 20:27:16.045758 19986 raft_consensus.cc:2454] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77af88fddd76447ead6eba1718e8755e in term 0.
I20260620 20:27:16.045881 19718 raft_consensus.cc:2454] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77af88fddd76447ead6eba1718e8755e in term 0.
I20260620 20:27:16.046438 19788 leader_election.cc:302] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77af88fddd76447ead6eba1718e8755e, ed36b3dd6b804e14ac55e671b21dc84b; no voters:
I20260620 20:27:16.046868 20056 raft_consensus.cc:2781] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:27:16.047036 20056 raft_consensus.cc:482] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:27:16.047166 20056 raft_consensus.cc:3037] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:16.049351 20056 raft_consensus.cc:504] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77af88fddd76447ead6eba1718e8755e" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34915 } } peers { permanent_uuid: "ed36b3dd6b804e14ac55e671b21dc84b" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35995 } } peers { permanent_uuid: "0c35137799034ebab3db71ae171a48db" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37153 } }
I20260620 20:27:16.050024 20056 leader_election.cc:288] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e [CANDIDATE]: Term 1 election: Requested vote from peers ed36b3dd6b804e14ac55e671b21dc84b (127.31.235.131:35995), 0c35137799034ebab3db71ae171a48db (127.31.235.129:37153)
I20260620 20:27:16.050570 19718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2443cba3e945ddb587c52b1f01bbad" candidate_uuid: "77af88fddd76447ead6eba1718e8755e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c35137799034ebab3db71ae171a48db"
I20260620 20:27:16.050568 19986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d2443cba3e945ddb587c52b1f01bbad" candidate_uuid: "77af88fddd76447ead6eba1718e8755e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed36b3dd6b804e14ac55e671b21dc84b"
I20260620 20:27:16.050787 19718 raft_consensus.cc:3037] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:16.050787 19986 raft_consensus.cc:3037] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:16.054047 19986 raft_consensus.cc:2454] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77af88fddd76447ead6eba1718e8755e in term 1.
I20260620 20:27:16.054047 19718 raft_consensus.cc:2454] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77af88fddd76447ead6eba1718e8755e in term 1.
I20260620 20:27:16.054457 19788 leader_election.cc:302] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77af88fddd76447ead6eba1718e8755e, ed36b3dd6b804e14ac55e671b21dc84b; no voters:
I20260620 20:27:16.054740 20056 raft_consensus.cc:2781] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:16.055209 20056 raft_consensus.cc:686] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e [term 1 LEADER]: Becoming Leader. State: Replica: 77af88fddd76447ead6eba1718e8755e, State: Running, Role: LEADER
I20260620 20:27:16.055743 20056 consensus_queue.cc:237] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77af88fddd76447ead6eba1718e8755e" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34915 } } peers { permanent_uuid: "ed36b3dd6b804e14ac55e671b21dc84b" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35995 } } peers { permanent_uuid: "0c35137799034ebab3db71ae171a48db" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37153 } }
I20260620 20:27:16.060514 19574 catalog_manager.cc:5495] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e reported cstate change: term changed from 0 to 1, leader changed from <none> to 77af88fddd76447ead6eba1718e8755e (127.31.235.130). New cstate: current_term: 1 leader_uuid: "77af88fddd76447ead6eba1718e8755e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77af88fddd76447ead6eba1718e8755e" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 34915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed36b3dd6b804e14ac55e671b21dc84b" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c35137799034ebab3db71ae171a48db" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37153 } health_report { overall_health: UNKNOWN } } }
I20260620 20:27:16.111513 20045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:16.105946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54430 (local address 127.31.235.130:34915)
0620 20:27:16.106153 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:16.106160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:16.106260 (+ 100us) server_negotiation.cc:407] Connection header received
0620 20:27:16.106513 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:16.106518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:16.106597 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:16.106759 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:27:16.107158 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:16.107900 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:16.109302 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:16.109745 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:16.109886 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:16.110388 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:27:16.110898 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:27:16.111068 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:16.111126 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 20:27:16.111211 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:27:16.120002 19986 raft_consensus.cc:1261] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b [term 1 FOLLOWER]: Refusing update from remote peer 77af88fddd76447ead6eba1718e8755e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:27:16.120002 19718 raft_consensus.cc:1261] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db [term 1 FOLLOWER]: Refusing update from remote peer 77af88fddd76447ead6eba1718e8755e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:27:16.121085 20061 consensus_queue.cc:1035] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed36b3dd6b804e14ac55e671b21dc84b" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:27:16.121462 20056 consensus_queue.cc:1035] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c35137799034ebab3db71ae171a48db" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 37153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:27:16.136270 20064 mvcc.cc:205] Tried to move back new op lower bound from 7299019719136935936 to 7299019718888177664. Current Snapshot: MvccSnapshot[applied={T|T < 7299019719136935936}]
I20260620 20:27:16.139901 20065 mvcc.cc:205] Tried to move back new op lower bound from 7299019719136935936 to 7299019718888177664. Current Snapshot: MvccSnapshot[applied={T|T < 7299019719136935936}]
I20260620 20:27:16.152910 19572 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:27:16.153258 19572 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:27:16.157292 19572 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b46a1646e6614cb584877d216f30dbb0: Sending DeleteTablet for 3 replicas of tablet 0d2443cba3e945ddb587c52b1f01bbad
I20260620 20:27:16.158465 19832 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d2443cba3e945ddb587c52b1f01bbad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:27:16 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:54418
I20260620 20:27:16.158586 19698 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d2443cba3e945ddb587c52b1f01bbad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:27:16 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:46776
I20260620 20:27:16.159181 19966 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d2443cba3e945ddb587c52b1f01bbad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:27:16 UTC) from {username='kudu', principal='kudu/127.31.235.190@KRBTEST.COM'} at 127.0.0.1:41990
I20260620 20:27:16.159516 20073 tablet_replica.cc:330] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e: stopping tablet replica
I20260620 20:27:16.160056 20073 raft_consensus.cc:2229] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:27:16.161041 20073 raft_consensus.cc:2258] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:27:16.162150 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 19635
I20260620 20:27:16.163872 20073 ts_tablet_manager.cc:1905] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:27:16.165992 20075 tablet_replica.cc:330] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b: stopping tablet replica
I20260620 20:27:16.166363 20075 raft_consensus.cc:2229] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:27:16.166795 20075 raft_consensus.cc:2258] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:27:16.170733 20074 tablet_replica.cc:330] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db: stopping tablet replica
I20260620 20:27:16.171253 20074 raft_consensus.cc:2229] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:27:16.171811 20074 raft_consensus.cc:2258] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:27:16.174029 20075 ts_tablet_manager.cc:1905] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:27:16.174028 20074 ts_tablet_manager.cc:1905] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 20:27:16 dist-test-slave-zzwf krb5kdc[19529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.129@KRBTEST.COM
I20260620 20:27:16.181049 20073 ts_tablet_manager.cc:1918] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:27:16.181217 20074 ts_tablet_manager.cc:1918] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:27:16.181341 20073 log.cc:1196] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/0d2443cba3e945ddb587c52b1f01bbad
I20260620 20:27:16.181450 20074 log.cc:1196] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-0/wal/wals/0d2443cba3e945ddb587c52b1f01bbad
I20260620 20:27:16.181442 20075 ts_tablet_manager.cc:1918] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:27:16.181675 20075 log.cc:1196] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/0d2443cba3e945ddb587c52b1f01bbad
I20260620 20:27:16.182085 20074 ts_tablet_manager.cc:1939] T 0d2443cba3e945ddb587c52b1f01bbad P 0c35137799034ebab3db71ae171a48db: Deleting consensus metadata
I20260620 20:27:16.182091 20073 ts_tablet_manager.cc:1939] T 0d2443cba3e945ddb587c52b1f01bbad P 77af88fddd76447ead6eba1718e8755e: Deleting consensus metadata
I20260620 20:27:16.182335 20075 ts_tablet_manager.cc:1939] T 0d2443cba3e945ddb587c52b1f01bbad P ed36b3dd6b804e14ac55e671b21dc84b: Deleting consensus metadata
I20260620 20:27:16.184062 19561 catalog_manager.cc:4841] TS ed36b3dd6b804e14ac55e671b21dc84b (127.31.235.131:35995): tablet 0d2443cba3e945ddb587c52b1f01bbad (table test-table [id=6f9cd175eeed475da6caf22d53e2d4ff]) successfully deleted
I20260620 20:27:16.184154 19558 catalog_manager.cc:4841] TS 0c35137799034ebab3db71ae171a48db (127.31.235.129:37153): tablet 0d2443cba3e945ddb587c52b1f01bbad (table test-table [id=6f9cd175eeed475da6caf22d53e2d4ff]) successfully deleted
I20260620 20:27:16.184459 19561 catalog_manager.cc:4841] TS 77af88fddd76447ead6eba1718e8755e (127.31.235.130:34915): tablet 0d2443cba3e945ddb587c52b1f01bbad (table test-table [id=6f9cd175eeed475da6caf22d53e2d4ff]) successfully deleted
I20260620 20:27:16.192072 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:16.163549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46788 (local address 127.31.235.129:37153)
0620 20:27:16.163729 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:16.163735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:16.170070 (+ 6335us) server_negotiation.cc:407] Connection header received
0620 20:27:16.170254 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:16.170261 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:16.170360 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:16.170483 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:16.172383 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:16.173601 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:16.175590 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:16.176298 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:16.176404 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:16.185754 (+ 9350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:16.185766 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:16.185781 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:16.185817 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:16.188843 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:16.189790 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:16.189798 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:16.189800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:16.189970 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:16.190446 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:16.190451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:16.190453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:16.190879 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:16.191144 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:16.191539 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 20:27:16.191712 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:27:16.192854 19758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:16.298552 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 19635
I20260620 20:27:16.316535 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 19769
Jun 20 20:27:16 dist-test-slave-zzwf krb5kdc[19529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.130@KRBTEST.COM
I20260620 20:27:16.333539 20045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:16.317411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.31.235.130:34915)
0620 20:27:16.317638 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:16.317647 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:16.317669 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:27:16.317741 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:16.317745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:16.317829 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:16.317937 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:16.319164 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:16.320008 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:16.321405 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:16.322258 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:16.322396 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:16.326905 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:16.326922 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:16.326943 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:16.327001 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:16.330794 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:16.331442 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:16.331451 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:16.331455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:16.331628 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:16.331976 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:16.331981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:16.331985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:16.332403 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:16.332741 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:16.332905 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 20:27:16.333129 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 20:27:16.334295 19892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:16.449012 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 19769
I20260620 20:27:16.467671 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 19903
Jun 20 20:27:16 dist-test-slave-zzwf krb5kdc[19529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.131@KRBTEST.COM
I20260620 20:27:16.482964 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:16.468585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.31.235.131:35995)
0620 20:27:16.468774 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:16.468779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:16.468815 (+ 36us) server_negotiation.cc:407] Connection header received
0620 20:27:16.469002 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:16.469006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:16.469075 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:16.469178 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:16.470470 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:16.471197 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:16.472620 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:16.473031 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:16.473129 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:16.477747 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:16.477769 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:16.477785 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:16.477819 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:16.480348 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:16.480968 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:16.480975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:16.480977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:16.481118 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:16.481548 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:16.481556 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:16.481561 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:16.481931 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:16.482157 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:16.482410 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 20:27:16.482578 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 20:27:16.483685 20026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:16.575947 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 19903
I20260620 20:27:16.592648 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 19542
I20260620 20:27:16.593845 19604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:16.704660 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 19542
2026-06-20T20:27:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4421 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T20:27:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:16Z Disabled control of system clock
I20260620 20:27:16.784642 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:43483
--webserver_interface=127.31.235.190
--webserver_port=0
--builtin_ntp_servers=127.31.235.148:40085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:43483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:27:17.122529 20084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:17.122848 20084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:17.122932 20084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:17.130898 20084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:17.131034 20084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:17.131074 20084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:17.131125 20084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:17.131171 20084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:17.140805 20084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:40085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:43483
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:43483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:17.142103 20084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:17.143800 20084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:17.155602 20092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:17.155664 20090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:17.155642 20089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:17.156487 20084 server_base.cc:1034] running on GCE node
I20260620 20:27:17.157225 20084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:17.158676 20084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:17.159919 20084 hybrid_clock.cc:648] HybridClock initialized: now 1781987237159870 us; error 57 us; skew 500 ppm
I20260620 20:27:17.160358 20084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:27:17.163040 20084 webserver.cc:460] Webserver started at http://127.31.235.190:37393/ using document root <none> and password file <none>
I20260620 20:27:17.163906 20084 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:17.164063 20084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:17.164431 20084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:17.167598 20084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "cdde85a2cc8d4424abea062224c57c93"
format_stamp: "Formatted at 2026-06-20 20:27:17 on dist-test-slave-zzwf"
server_key: "07d42265f3a5128e0b3ec87652b5ec57"
server_key_iv: "7ca17030b2732e9ce55bd2b52f4d40ad"
server_key_version: "encryptionkey@0"
I20260620 20:27:17.168476 20084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "cdde85a2cc8d4424abea062224c57c93"
format_stamp: "Formatted at 2026-06-20 20:27:17 on dist-test-slave-zzwf"
server_key: "07d42265f3a5128e0b3ec87652b5ec57"
server_key_iv: "7ca17030b2732e9ce55bd2b52f4d40ad"
server_key_version: "encryptionkey@0"
I20260620 20:27:17.174007 20084 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:27:17.177512 20098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:17.178830 20084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:27:17.179029 20084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "cdde85a2cc8d4424abea062224c57c93"
format_stamp: "Formatted at 2026-06-20 20:27:17 on dist-test-slave-zzwf"
server_key: "07d42265f3a5128e0b3ec87652b5ec57"
server_key_iv: "7ca17030b2732e9ce55bd2b52f4d40ad"
server_key_version: "encryptionkey@0"
I20260620 20:27:17.179260 20084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:17.208822 20084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:17.209805 20084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:17.210098 20084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:17.229122 20084 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:43483
I20260620 20:27:17.229141 20149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:43483 every 8 connection(s)
I20260620 20:27:17.230537 20084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:17.234257 20150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:17.240082 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 20084
I20260620 20:27:17.240293 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:17.240775 32686 external_mini_cluster.cc:1428] Setting key 2dfe084fd98f38a42114e25c789fc67d
I20260620 20:27:17.245421 20150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93: Bootstrap starting.
I20260620 20:27:17.249604 20150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:17.250738 20150 log.cc:824] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:17.253892 20150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93: No bootstrap required, opened a new log
I20260620 20:27:17.256680 20153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:17.242791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38686 (local address 127.31.235.190:43483)
0620 20:27:17.243832 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:17.243848 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:17.243898 (+ 50us) server_negotiation.cc:407] Connection header received
0620 20:27:17.246310 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:17.246336 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:17.246879 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:17.247358 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:17.248022 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:17.250017 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:17.251619 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:17.252480 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:17.252667 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:17.253833 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:17.253843 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:17.253862 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:17.254577 (+ 715us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:17.254677 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:17.254704 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:17.254835 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:17.255067 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 20:27:17.255435 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":327,"threads_started":1}
I20260620 20:27:17.262881 20150 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdde85a2cc8d4424abea062224c57c93" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 43483 } }
I20260620 20:27:17.263221 20150 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:17.263283 20150 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdde85a2cc8d4424abea062224c57c93, State: Initialized, Role: FOLLOWER
I20260620 20:27:17.263824 20150 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdde85a2cc8d4424abea062224c57c93" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 43483 } }
I20260620 20:27:17.264045 20150 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:17.264142 20150 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:17.264267 20150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:17.266700 20150 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdde85a2cc8d4424abea062224c57c93" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 43483 } }
I20260620 20:27:17.267149 20150 leader_election.cc:302] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdde85a2cc8d4424abea062224c57c93; no voters:
I20260620 20:27:17.267728 20150 leader_election.cc:288] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:17.267935 20155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:17.268798 20155 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93 [term 1 LEADER]: Becoming Leader. State: Replica: cdde85a2cc8d4424abea062224c57c93, State: Running, Role: LEADER
I20260620 20:27:17.269367 20155 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdde85a2cc8d4424abea062224c57c93" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 43483 } }
I20260620 20:27:17.269922 20150 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:17.272102 20157 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdde85a2cc8d4424abea062224c57c93. Latest consensus state: current_term: 1 leader_uuid: "cdde85a2cc8d4424abea062224c57c93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdde85a2cc8d4424abea062224c57c93" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 43483 } } }
I20260620 20:27:17.272652 20157 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:17.272583 20156 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdde85a2cc8d4424abea062224c57c93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdde85a2cc8d4424abea062224c57c93" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 43483 } } }
I20260620 20:27:17.272892 20156 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:17.274858 20160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:17.280542 20160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:17.290158 20160 catalog_manager.cc:1347] Generated new cluster ID: 048fe9ea5d844e9e98e4e15c2d1ca78f
I20260620 20:27:17.290297 20160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:17.298401 20160 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:17.299681 20160 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:17.307631 20160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93: Generated new TSK 0
I20260620 20:27:17.308459 20160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:17.318467 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.129:0
--local_ip_for_outbound_sockets=127.31.235.129
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:43483
--builtin_ntp_servers=127.31.235.148:40085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 20:27:17.654397 20174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:17.654663 20174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:17.654796 20174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:17.662156 20174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:17.662292 20174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:17.662472 20174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.129
I20260620 20:27:17.671880 20174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:40085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.235.129
--webserver_port=0
--tserver_master_addrs=127.31.235.190:43483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.129
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:17.673185 20174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:17.675021 20174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:17.689270 20179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:17.689335 20182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:17.689358 20180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:17.690361 20174 server_base.cc:1034] running on GCE node
I20260620 20:27:17.690984 20174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:17.691957 20174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:17.693200 20174 hybrid_clock.cc:648] HybridClock initialized: now 1781987237693116 us; error 53 us; skew 500 ppm
I20260620 20:27:17.693570 20174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:27:17.696091 20174 webserver.cc:460] Webserver started at http://127.31.235.129:33373/ using document root <none> and password file <none>
I20260620 20:27:17.696841 20174 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:17.696935 20174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:17.697211 20174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:17.700326 20174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-0/data/instance:
uuid: "f3772f32d8a14f7882b2af2d4cc30be6"
format_stamp: "Formatted at 2026-06-20 20:27:17 on dist-test-slave-zzwf"
server_key: "7494958d67163a8b5b1b372ce5291a28"
server_key_iv: "58f7b1055645b5820400f2dfc7337422"
server_key_version: "encryptionkey@0"
I20260620 20:27:17.701130 20174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance:
uuid: "f3772f32d8a14f7882b2af2d4cc30be6"
format_stamp: "Formatted at 2026-06-20 20:27:17 on dist-test-slave-zzwf"
server_key: "7494958d67163a8b5b1b372ce5291a28"
server_key_iv: "58f7b1055645b5820400f2dfc7337422"
server_key_version: "encryptionkey@0"
I20260620 20:27:17.706769 20174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 20:27:17.710481 20188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:17.711915 20174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:27:17.712134 20174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-0/wal
uuid: "f3772f32d8a14f7882b2af2d4cc30be6"
format_stamp: "Formatted at 2026-06-20 20:27:17 on dist-test-slave-zzwf"
server_key: "7494958d67163a8b5b1b372ce5291a28"
server_key_iv: "58f7b1055645b5820400f2dfc7337422"
server_key_version: "encryptionkey@0"
I20260620 20:27:17.712400 20174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:17.727202 20174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:17.728149 20174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:17.728499 20174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:17.729593 20174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:27:17.731143 20174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:27:17.731277 20174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:17.731433 20174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:27:17.731534 20174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:17.767379 20174 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.129:41225
I20260620 20:27:17.767441 20300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.129:41225 every 8 connection(s)
I20260620 20:27:17.768791 20174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-0/data/info.pb
I20260620 20:27:17.777190 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 20174
I20260620 20:27:17.777411 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-0/wal/instance
I20260620 20:27:17.777984 32686 external_mini_cluster.cc:1428] Setting key 5ebebfa74d3c10a171311d06cf033002
I20260620 20:27:17.781265 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.130:0
--local_ip_for_outbound_sockets=127.31.235.130
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:43483
--builtin_ntp_servers=127.31.235.148:40085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:27:17.782912 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:17.771703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.129:49585 (local address 127.31.235.190:43483)
0620 20:27:17.772192 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:17.772199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:17.773257 (+ 1058us) server_negotiation.cc:407] Connection header received
0620 20:27:17.775114 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:17.775120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:17.775213 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:17.775332 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:17.776487 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:17.777293 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:17.779721 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:17.780399 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:17.780528 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:17.782060 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:17.782066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:17.782071 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:17.782098 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:17.782138 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:17.782150 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:17.782249 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:17.782487 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 20:27:17.782586 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":258,"threads_started":1}
I20260620 20:27:17.783970 20304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:17.772204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:43483 (local address 127.31.235.129:49585)
0620 20:27:17.773086 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:17.773119 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:17.774820 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:17.775617 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:17.775626 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:17.775714 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:17.776272 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:17.776288 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:17.777548 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:17.777555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:17.779576 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:17.779592 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:17.780620 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:17.780630 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:17.780888 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:17.781728 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:27:17.781731 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:17.781774 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:27:17.781863 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:17.782247 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:17.782261 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:17.782272 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 20:27:17.782464 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 20:27:17.782891 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":279,"threads_started":1}
I20260620 20:27:17.785476 20301 heartbeater.cc:347] Connected to a master server at 127.31.235.190:43483
I20260620 20:27:17.785846 20301 heartbeater.cc:464] Registering TS with master...
I20260620 20:27:17.786778 20301 heartbeater.cc:511] Master 127.31.235.190:43483 requested a full tablet report, sending...
I20260620 20:27:17.789166 20115 ts_manager.cc:194] Registered new tserver with Master: f3772f32d8a14f7882b2af2d4cc30be6 (127.31.235.129:41225)
I20260620 20:27:17.790911 20115 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.235.129:49585
W20260620 20:27:18.114065 20305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:18.114383 20305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:18.114569 20305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:18.122370 20305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:18.122488 20305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:18.122689 20305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.130
I20260620 20:27:18.132177 20305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:40085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.235.130
--webserver_port=0
--tserver_master_addrs=127.31.235.190:43483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.130
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:18.133539 20305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:18.135399 20305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:18.148856 20311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:18.148800 20313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:18.148830 20310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:18.149823 20305 server_base.cc:1034] running on GCE node
I20260620 20:27:18.150529 20305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:18.151757 20305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:18.153014 20305 hybrid_clock.cc:648] HybridClock initialized: now 1781987238152972 us; error 69 us; skew 500 ppm
I20260620 20:27:18.153360 20305 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:27:18.155663 20305 webserver.cc:460] Webserver started at http://127.31.235.130:38621/ using document root <none> and password file <none>
I20260620 20:27:18.156369 20305 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:18.156499 20305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:18.156805 20305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:18.159653 20305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-1/data/instance:
uuid: "fdb3f7bbfd3b4e2bb7d1cf36fb15a98f"
format_stamp: "Formatted at 2026-06-20 20:27:18 on dist-test-slave-zzwf"
server_key: "7f2d694d7c5742226b604096c0eeb2b0"
server_key_iv: "e6eaa1c01a50d49bd6eac2c477cafe2b"
server_key_version: "encryptionkey@0"
I20260620 20:27:18.160434 20305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance:
uuid: "fdb3f7bbfd3b4e2bb7d1cf36fb15a98f"
format_stamp: "Formatted at 2026-06-20 20:27:18 on dist-test-slave-zzwf"
server_key: "7f2d694d7c5742226b604096c0eeb2b0"
server_key_iv: "e6eaa1c01a50d49bd6eac2c477cafe2b"
server_key_version: "encryptionkey@0"
I20260620 20:27:18.165685 20305 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 20:27:18.169189 20319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:18.170630 20305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:27:18.170862 20305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-1/wal
uuid: "fdb3f7bbfd3b4e2bb7d1cf36fb15a98f"
format_stamp: "Formatted at 2026-06-20 20:27:18 on dist-test-slave-zzwf"
server_key: "7f2d694d7c5742226b604096c0eeb2b0"
server_key_iv: "e6eaa1c01a50d49bd6eac2c477cafe2b"
server_key_version: "encryptionkey@0"
I20260620 20:27:18.171204 20305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:18.191154 20305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:18.192087 20305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:18.192420 20305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:18.193660 20305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:27:18.195396 20305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:27:18.195487 20305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:18.195560 20305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:27:18.195621 20305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:18.230068 20305 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.130:43917
I20260620 20:27:18.230109 20431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.130:43917 every 8 connection(s)
I20260620 20:27:18.231715 20305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-1/data/info.pb
I20260620 20:27:18.239475 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 20305
I20260620 20:27:18.239679 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-1/wal/instance
I20260620 20:27:18.240295 32686 external_mini_cluster.cc:1428] Setting key 55074367567d6808414a6abceac4989a
I20260620 20:27:18.243539 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.235.131:0
--local_ip_for_outbound_sockets=127.31.235.131
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:43483
--builtin_ntp_servers=127.31.235.148:40085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 20:27:18.250232 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:18.235351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:55845 (local address 127.31.235.190:43483)
0620 20:27:18.235568 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:18.235573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:18.237086 (+ 1513us) server_negotiation.cc:407] Connection header received
0620 20:27:18.239227 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:18.239234 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:18.239323 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:18.239429 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:18.240968 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.242076 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.244732 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.245383 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.245544 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:18.248927 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:18.248935 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:18.248942 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:18.248968 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:18.249043 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:18.249087 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:18.249187 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:18.249670 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 20:27:18.249796 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:27:18.251968 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:18.235867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:43483 (local address 127.31.235.130:55845)
0620 20:27:18.236900 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:18.236946 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:18.238878 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:18.239749 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:18.239763 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:18.239898 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:18.240785 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:18.240811 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.242298 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.242302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:18.244566 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:18.244585 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.245668 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.245678 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:18.246038 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:18.248256 (+ 2218us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:27:18.248287 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:18.248378 (+ 91us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:27:18.248551 (+ 173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:18.249307 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:18.249346 (+ 39us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:18.249357 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 20:27:18.249654 (+ 297us) client_negotiation.cc:240] Negotiation successful
0620 20:27:18.250384 (+ 730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":342,"threads_started":1}
I20260620 20:27:18.254137 20432 heartbeater.cc:347] Connected to a master server at 127.31.235.190:43483
I20260620 20:27:18.254539 20432 heartbeater.cc:464] Registering TS with master...
I20260620 20:27:18.255578 20432 heartbeater.cc:511] Master 127.31.235.190:43483 requested a full tablet report, sending...
I20260620 20:27:18.257385 20115 ts_manager.cc:194] Registered new tserver with Master: fdb3f7bbfd3b4e2bb7d1cf36fb15a98f (127.31.235.130:43917)
I20260620 20:27:18.258348 20115 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.235.130:55845
W20260620 20:27:18.584165 20435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:18.584466 20435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:18.584604 20435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:18.591938 20435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:18.592051 20435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:18.592233 20435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.235.131
I20260620 20:27:18.601421 20435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:40085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.235.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.235.131
--webserver_port=0
--tserver_master_addrs=127.31.235.190:43483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.235.131
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:18.602866 20435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:18.604712 20435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:18.618345 20441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:18.618433 20440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:18.618387 20443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:18.619377 20435 server_base.cc:1034] running on GCE node
I20260620 20:27:18.620014 20435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:18.620997 20435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:18.622241 20435 hybrid_clock.cc:648] HybridClock initialized: now 1781987238622189 us; error 64 us; skew 500 ppm
I20260620 20:27:18.622570 20435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:27:18.624996 20435 webserver.cc:460] Webserver started at http://127.31.235.131:34657/ using document root <none> and password file <none>
I20260620 20:27:18.625877 20435 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:18.625993 20435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:18.626305 20435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:18.629196 20435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-2/data/instance:
uuid: "9e3d27ef13024e55a57d62de7cf93cd1"
format_stamp: "Formatted at 2026-06-20 20:27:18 on dist-test-slave-zzwf"
server_key: "1c0a525fad4dbd005ff3f77637c6d331"
server_key_iv: "b528c7b3d3c0ba86d9c1fdc279a6e98e"
server_key_version: "encryptionkey@0"
I20260620 20:27:18.630023 20435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance:
uuid: "9e3d27ef13024e55a57d62de7cf93cd1"
format_stamp: "Formatted at 2026-06-20 20:27:18 on dist-test-slave-zzwf"
server_key: "1c0a525fad4dbd005ff3f77637c6d331"
server_key_iv: "b528c7b3d3c0ba86d9c1fdc279a6e98e"
server_key_version: "encryptionkey@0"
I20260620 20:27:18.635142 20435 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:27:18.638787 20449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:18.640269 20435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:27:18.640486 20435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-2/wal
uuid: "9e3d27ef13024e55a57d62de7cf93cd1"
format_stamp: "Formatted at 2026-06-20 20:27:18 on dist-test-slave-zzwf"
server_key: "1c0a525fad4dbd005ff3f77637c6d331"
server_key_iv: "b528c7b3d3c0ba86d9c1fdc279a6e98e"
server_key_version: "encryptionkey@0"
I20260620 20:27:18.640722 20435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:18.652436 20435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:18.653244 20435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:18.653649 20435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:18.654603 20435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 20:27:18.656073 20435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 20:27:18.656205 20435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:18.656337 20435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 20:27:18.656426 20435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:18.691699 20435 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.131:35105
I20260620 20:27:18.691792 20561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.131:35105 every 8 connection(s)
I20260620 20:27:18.693029 20435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-2/data/info.pb
I20260620 20:27:18.694432 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 20435
I20260620 20:27:18.694619 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-2/wal/instance
I20260620 20:27:18.695065 32686 external_mini_cluster.cc:1428] Setting key 362078758767972a75d9dd5c1decf91b
I20260620 20:27:18.706282 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:18.695769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.131:54477 (local address 127.31.235.190:43483)
0620 20:27:18.696002 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:18.696008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:18.697841 (+ 1833us) server_negotiation.cc:407] Connection header received
0620 20:27:18.699615 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:18.699619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:18.699689 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:18.699776 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:18.700842 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.701715 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.703377 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.703781 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.703884 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:18.705356 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:18.705361 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:18.705364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:18.705392 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:18.705423 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:18.705436 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:18.705578 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:18.705959 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 20:27:18.706045 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:27:18.707393 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:18.696457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:43483 (local address 127.31.235.131:54477)
0620 20:27:18.697683 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:18.697730 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:18.699330 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:18.699944 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:18.699952 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:18.700054 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:18.700652 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:18.700669 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.701881 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.701886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:18.703203 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:18.703217 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.704022 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.704029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:18.704265 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:18.705000 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 20:27:18.705002 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:18.705051 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 20:27:18.705133 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:18.705627 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:18.705640 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:18.705652 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 20:27:18.705958 (+ 306us) client_negotiation.cc:240] Negotiation successful
0620 20:27:18.706292 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":430,"threads_started":1}
I20260620 20:27:18.709134 20562 heartbeater.cc:347] Connected to a master server at 127.31.235.190:43483
I20260620 20:27:18.709547 20562 heartbeater.cc:464] Registering TS with master...
I20260620 20:27:18.710353 20562 heartbeater.cc:511] Master 127.31.235.190:43483 requested a full tablet report, sending...
I20260620 20:27:18.711932 20115 ts_manager.cc:194] Registered new tserver with Master: 9e3d27ef13024e55a57d62de7cf93cd1 (127.31.235.131:35105)
I20260620 20:27:18.712747 20115 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.235.131:54477
I20260620 20:27:18.723179 32686 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 20:27:18.733325 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:18.727380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38694 (local address 127.31.235.190:43483)
0620 20:27:18.727642 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:18.727651 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:18.728061 (+ 410us) server_negotiation.cc:407] Connection header received
0620 20:27:18.728256 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:18.728260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:18.728347 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:18.728437 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:18.728896 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.729690 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.731046 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.731446 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.731556 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:18.732619 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:18.732625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:18.732628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:18.732655 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:18.732686 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:18.732698 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:18.732771 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:18.732888 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 20:27:18.732974 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 20:27:18.737282 20115 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 20:27:18.744376 20115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 20:27:18.768924 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:18.757998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45292 (local address 127.31.235.131:35105)
0620 20:27:18.758757 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:18.758765 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:18.759720 (+ 955us) server_negotiation.cc:407] Connection header received
0620 20:27:18.759931 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:18.759941 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:18.760173 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:18.760338 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:18.761381 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.763036 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.765901 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.767302 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.767479 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:18.767712 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:18.768365 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 20:27:18.768537 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":467,"threads_started":1}
I20260620 20:27:18.769796 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:18.757431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:35105 (local address 127.0.0.1:45292)
0620 20:27:18.759595 (+ 2164us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:18.759613 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:18.759718 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:18.760664 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:18.760673 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:18.760704 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:18.761141 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:18.761154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.763222 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.763228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:18.765668 (+ 2440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:18.765683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.767868 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.767877 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:18.768082 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:18.768140 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 20:27:18.769279 (+ 1139us) client_negotiation.cc:240] Negotiation successful
0620 20:27:18.769505 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2047,"thread_start_us":1512,"threads_started":1}
I20260620 20:27:18.771651 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:18.757437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45234 (local address 127.31.235.129:41225)
0620 20:27:18.758326 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:18.758335 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:18.759520 (+ 1185us) server_negotiation.cc:407] Connection header received
0620 20:27:18.759956 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:18.759963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:18.760173 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:18.760338 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:18.762265 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.763710 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.767799 (+ 4089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.769391 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.769574 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:18.769771 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:18.770874 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0620 20:27:18.771094 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":820,"threads_started":1}
I20260620 20:27:18.772689 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:18.757361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:41225 (local address 127.0.0.1:45234)
0620 20:27:18.759348 (+ 1987us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:18.759403 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:18.759683 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:18.761692 (+ 2009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:18.761698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:18.761723 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:18.762088 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:18.762099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.763900 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.763905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:18.767535 (+ 3630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:18.767549 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.770466 (+ 2917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.770471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:18.770658 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:18.770688 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 20:27:18.772040 (+ 1352us) client_negotiation.cc:240] Negotiation successful
0620 20:27:18.772247 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1696,"thread_start_us":1942,"threads_started":1}
I20260620 20:27:18.777067 20236 tablet_service.cc:1467] Processing CreateTablet for tablet edf22a12b03045f29153b49d55351203 (DEFAULT_TABLE table=test-table [id=7aa21c66e2d7436282fe3c5c43deb1e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:27:18.778661 20236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edf22a12b03045f29153b49d55351203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:18.779271 20497 tablet_service.cc:1467] Processing CreateTablet for tablet edf22a12b03045f29153b49d55351203 (DEFAULT_TABLE table=test-table [id=7aa21c66e2d7436282fe3c5c43deb1e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:27:18.780293 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:18.759837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.130:43917 (local address 127.0.0.1:58958)
0620 20:27:18.770119 (+ 10282us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:18.770138 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:18.770224 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:18.774570 (+ 4346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:18.774576 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:18.774605 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:18.774787 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:18.774799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.776854 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.776859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:18.778173 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:18.778192 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.779513 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.779521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:18.779693 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:18.779723 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 20:27:18.779843 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 20:27:18.779993 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10161,"thread_start_us":14494,"threads_started":1}
I20260620 20:27:18.780386 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:18.760125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58958 (local address 127.31.235.130:43917)
0620 20:27:18.773511 (+ 13386us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:18.773524 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:18.773564 (+ 40us) server_negotiation.cc:407] Connection header received
0620 20:27:18.773684 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:18.773697 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:18.774011 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:18.774994 (+ 983us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:18.775212 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.776676 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.778332 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.779330 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.779512 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:18.779722 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:18.779882 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 20:27:18.780038 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13120,"thread_start_us":16469,"threads_started":1}
I20260620 20:27:18.780964 20497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edf22a12b03045f29153b49d55351203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:18.784101 20367 tablet_service.cc:1467] Processing CreateTablet for tablet edf22a12b03045f29153b49d55351203 (DEFAULT_TABLE table=test-table [id=7aa21c66e2d7436282fe3c5c43deb1e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 20:27:18.785302 20367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edf22a12b03045f29153b49d55351203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:18.793859 20301 heartbeater.cc:503] Master 127.31.235.190:43483 was elected leader, sending a full tablet report...
I20260620 20:27:18.795301 20580 tablet_bootstrap.cc:492] T edf22a12b03045f29153b49d55351203 P f3772f32d8a14f7882b2af2d4cc30be6: Bootstrap starting.
I20260620 20:27:18.797729 20581 tablet_bootstrap.cc:492] T edf22a12b03045f29153b49d55351203 P 9e3d27ef13024e55a57d62de7cf93cd1: Bootstrap starting.
I20260620 20:27:18.799113 20582 tablet_bootstrap.cc:492] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f: Bootstrap starting.
I20260620 20:27:18.799289 20580 tablet_bootstrap.cc:654] T edf22a12b03045f29153b49d55351203 P f3772f32d8a14f7882b2af2d4cc30be6: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:18.800436 20580 log.cc:824] T edf22a12b03045f29153b49d55351203 P f3772f32d8a14f7882b2af2d4cc30be6: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:18.801338 20581 tablet_bootstrap.cc:654] T edf22a12b03045f29153b49d55351203 P 9e3d27ef13024e55a57d62de7cf93cd1: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:18.802564 20581 log.cc:824] T edf22a12b03045f29153b49d55351203 P 9e3d27ef13024e55a57d62de7cf93cd1: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:18.802912 20582 tablet_bootstrap.cc:654] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:18.804332 20582 log.cc:824] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:18.807139 20580 tablet_bootstrap.cc:492] T edf22a12b03045f29153b49d55351203 P f3772f32d8a14f7882b2af2d4cc30be6: No bootstrap required, opened a new log
I20260620 20:27:18.807367 20582 tablet_bootstrap.cc:492] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f: No bootstrap required, opened a new log
I20260620 20:27:18.807355 20581 tablet_bootstrap.cc:492] T edf22a12b03045f29153b49d55351203 P 9e3d27ef13024e55a57d62de7cf93cd1: No bootstrap required, opened a new log
I20260620 20:27:18.807444 20580 ts_tablet_manager.cc:1397] T edf22a12b03045f29153b49d55351203 P f3772f32d8a14f7882b2af2d4cc30be6: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260620 20:27:18.807612 20582 ts_tablet_manager.cc:1397] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 20:27:18.807612 20581 ts_tablet_manager.cc:1397] T edf22a12b03045f29153b49d55351203 P 9e3d27ef13024e55a57d62de7cf93cd1: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 20:27:18.815510 20580 raft_consensus.cc:348] T edf22a12b03045f29153b49d55351203 P f3772f32d8a14f7882b2af2d4cc30be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3772f32d8a14f7882b2af2d4cc30be6" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41225 } } peers { permanent_uuid: "9e3d27ef13024e55a57d62de7cf93cd1" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35105 } } peers { permanent_uuid: "fdb3f7bbfd3b4e2bb7d1cf36fb15a98f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43917 } }
I20260620 20:27:18.815963 20580 raft_consensus.cc:374] T edf22a12b03045f29153b49d55351203 P f3772f32d8a14f7882b2af2d4cc30be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:18.816063 20580 raft_consensus.cc:729] T edf22a12b03045f29153b49d55351203 P f3772f32d8a14f7882b2af2d4cc30be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3772f32d8a14f7882b2af2d4cc30be6, State: Initialized, Role: FOLLOWER
I20260620 20:27:18.816740 20580 consensus_queue.cc:260] T edf22a12b03045f29153b49d55351203 P f3772f32d8a14f7882b2af2d4cc30be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3772f32d8a14f7882b2af2d4cc30be6" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41225 } } peers { permanent_uuid: "9e3d27ef13024e55a57d62de7cf93cd1" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35105 } } peers { permanent_uuid: "fdb3f7bbfd3b4e2bb7d1cf36fb15a98f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43917 } }
I20260620 20:27:18.818502 20582 raft_consensus.cc:348] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3772f32d8a14f7882b2af2d4cc30be6" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41225 } } peers { permanent_uuid: "9e3d27ef13024e55a57d62de7cf93cd1" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35105 } } peers { permanent_uuid: "fdb3f7bbfd3b4e2bb7d1cf36fb15a98f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43917 } }
I20260620 20:27:18.818502 20581 raft_consensus.cc:348] T edf22a12b03045f29153b49d55351203 P 9e3d27ef13024e55a57d62de7cf93cd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3772f32d8a14f7882b2af2d4cc30be6" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41225 } } peers { permanent_uuid: "9e3d27ef13024e55a57d62de7cf93cd1" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35105 } } peers { permanent_uuid: "fdb3f7bbfd3b4e2bb7d1cf36fb15a98f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43917 } }
I20260620 20:27:18.818938 20581 raft_consensus.cc:374] T edf22a12b03045f29153b49d55351203 P 9e3d27ef13024e55a57d62de7cf93cd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:18.818948 20582 raft_consensus.cc:374] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:18.819041 20582 raft_consensus.cc:729] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdb3f7bbfd3b4e2bb7d1cf36fb15a98f, State: Initialized, Role: FOLLOWER
I20260620 20:27:18.819113 20581 raft_consensus.cc:729] T edf22a12b03045f29153b49d55351203 P 9e3d27ef13024e55a57d62de7cf93cd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3d27ef13024e55a57d62de7cf93cd1, State: Initialized, Role: FOLLOWER
I20260620 20:27:18.819832 20581 consensus_queue.cc:260] T edf22a12b03045f29153b49d55351203 P 9e3d27ef13024e55a57d62de7cf93cd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3772f32d8a14f7882b2af2d4cc30be6" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41225 } } peers { permanent_uuid: "9e3d27ef13024e55a57d62de7cf93cd1" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35105 } } peers { permanent_uuid: "fdb3f7bbfd3b4e2bb7d1cf36fb15a98f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43917 } }
I20260620 20:27:18.819821 20582 consensus_queue.cc:260] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3772f32d8a14f7882b2af2d4cc30be6" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41225 } } peers { permanent_uuid: "9e3d27ef13024e55a57d62de7cf93cd1" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35105 } } peers { permanent_uuid: "fdb3f7bbfd3b4e2bb7d1cf36fb15a98f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43917 } }
I20260620 20:27:18.820401 20580 ts_tablet_manager.cc:1428] T edf22a12b03045f29153b49d55351203 P f3772f32d8a14f7882b2af2d4cc30be6: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20260620 20:27:18.822146 20562 heartbeater.cc:503] Master 127.31.235.190:43483 was elected leader, sending a full tablet report...
I20260620 20:27:18.822721 20581 ts_tablet_manager.cc:1428] T edf22a12b03045f29153b49d55351203 P 9e3d27ef13024e55a57d62de7cf93cd1: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260620 20:27:18.823144 20582 ts_tablet_manager.cc:1428] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260620 20:27:18.823658 20432 heartbeater.cc:503] Master 127.31.235.190:43483 was elected leader, sending a full tablet report...
I20260620 20:27:18.839041 20587 raft_consensus.cc:482] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 20:27:18.839411 20587 raft_consensus.cc:504] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3772f32d8a14f7882b2af2d4cc30be6" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41225 } } peers { permanent_uuid: "9e3d27ef13024e55a57d62de7cf93cd1" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35105 } } peers { permanent_uuid: "fdb3f7bbfd3b4e2bb7d1cf36fb15a98f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43917 } }
I20260620 20:27:18.841389 20587 leader_election.cc:288] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3772f32d8a14f7882b2af2d4cc30be6 (127.31.235.129:41225), 9e3d27ef13024e55a57d62de7cf93cd1 (127.31.235.131:35105)
I20260620 20:27:18.849148 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:18.842521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:60655 (local address 127.31.235.131:35105)
0620 20:27:18.842733 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:18.842741 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:18.843520 (+ 779us) server_negotiation.cc:407] Connection header received
0620 20:27:18.843838 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:18.843844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:18.843904 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:18.844001 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:18.844697 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.845484 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.847348 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.848175 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.848340 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:18.848417 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:18.848772 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 20:27:18.848901 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 20:27:18.849155 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:18.842018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.235.130:46543 (local address 127.31.235.129:41225)
0620 20:27:18.842290 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:18.842301 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:18.843510 (+ 1209us) server_negotiation.cc:407] Connection header received
0620 20:27:18.843751 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:18.843757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:18.843842 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:18.843941 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 20:27:18.844639 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.845394 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.847381 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.848173 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.848360 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:18.848439 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:18.848766 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 20:27:18.848897 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 20:27:18.849337 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:18.841890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.131:35105 (local address 127.31.235.130:60655)
0620 20:27:18.843412 (+ 1522us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:18.843428 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:18.843618 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:18.844013 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:18.844017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:18.844044 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:18.844525 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:18.844542 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.845672 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.845679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:18.847159 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:18.847170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.848396 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.848403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:18.848602 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:18.848639 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 20:27:18.848743 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 20:27:18.848897 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1400,"mutex_wait_us":76,"spinlock_wait_cycles":3328,"thread_start_us":1191,"threads_started":1}
I20260620 20:27:18.849550 20589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:18.841606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:41225 (local address 127.31.235.130:46543)
0620 20:27:18.843400 (+ 1794us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:18.843421 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:18.843588 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:18.843956 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:18.843960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:18.843983 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 20:27:18.844500 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:18.844518 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.845544 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.845552 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:18.847168 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:18.847179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.848416 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.848423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:18.848607 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:18.848635 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 20:27:18.848738 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 20:27:18.848896 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1625,"mutex_wait_us":11,"spinlock_wait_cycles":4480,"thread_start_us":1176,"threads_started":1}
I20260620 20:27:18.850257 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edf22a12b03045f29153b49d55351203" candidate_uuid: "fdb3f7bbfd3b4e2bb7d1cf36fb15a98f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e3d27ef13024e55a57d62de7cf93cd1" is_pre_election: true
I20260620 20:27:18.850504 20256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edf22a12b03045f29153b49d55351203" candidate_uuid: "fdb3f7bbfd3b4e2bb7d1cf36fb15a98f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3772f32d8a14f7882b2af2d4cc30be6" is_pre_election: true
I20260620 20:27:18.850770 20517 raft_consensus.cc:2454] T edf22a12b03045f29153b49d55351203 P 9e3d27ef13024e55a57d62de7cf93cd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdb3f7bbfd3b4e2bb7d1cf36fb15a98f in term 0.
I20260620 20:27:18.851042 20256 raft_consensus.cc:2454] T edf22a12b03045f29153b49d55351203 P f3772f32d8a14f7882b2af2d4cc30be6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdb3f7bbfd3b4e2bb7d1cf36fb15a98f in term 0.
I20260620 20:27:18.851444 20323 leader_election.cc:302] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e3d27ef13024e55a57d62de7cf93cd1, fdb3f7bbfd3b4e2bb7d1cf36fb15a98f; no voters:
I20260620 20:27:18.851788 20587 raft_consensus.cc:2781] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 20:27:18.851933 20587 raft_consensus.cc:482] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 20:27:18.852046 20587 raft_consensus.cc:3037] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:18.854270 20587 raft_consensus.cc:504] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3772f32d8a14f7882b2af2d4cc30be6" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41225 } } peers { permanent_uuid: "9e3d27ef13024e55a57d62de7cf93cd1" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35105 } } peers { permanent_uuid: "fdb3f7bbfd3b4e2bb7d1cf36fb15a98f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43917 } }
I20260620 20:27:18.854905 20587 leader_election.cc:288] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f [CANDIDATE]: Term 1 election: Requested vote from peers f3772f32d8a14f7882b2af2d4cc30be6 (127.31.235.129:41225), 9e3d27ef13024e55a57d62de7cf93cd1 (127.31.235.131:35105)
I20260620 20:27:18.855378 20256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edf22a12b03045f29153b49d55351203" candidate_uuid: "fdb3f7bbfd3b4e2bb7d1cf36fb15a98f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3772f32d8a14f7882b2af2d4cc30be6"
I20260620 20:27:18.855574 20256 raft_consensus.cc:3037] T edf22a12b03045f29153b49d55351203 P f3772f32d8a14f7882b2af2d4cc30be6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:18.855499 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edf22a12b03045f29153b49d55351203" candidate_uuid: "fdb3f7bbfd3b4e2bb7d1cf36fb15a98f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e3d27ef13024e55a57d62de7cf93cd1"
I20260620 20:27:18.855774 20517 raft_consensus.cc:3037] T edf22a12b03045f29153b49d55351203 P 9e3d27ef13024e55a57d62de7cf93cd1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:18.857897 20256 raft_consensus.cc:2454] T edf22a12b03045f29153b49d55351203 P f3772f32d8a14f7882b2af2d4cc30be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdb3f7bbfd3b4e2bb7d1cf36fb15a98f in term 1.
I20260620 20:27:18.858273 20517 raft_consensus.cc:2454] T edf22a12b03045f29153b49d55351203 P 9e3d27ef13024e55a57d62de7cf93cd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdb3f7bbfd3b4e2bb7d1cf36fb15a98f in term 1.
I20260620 20:27:18.858310 20320 leader_election.cc:302] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f3772f32d8a14f7882b2af2d4cc30be6, fdb3f7bbfd3b4e2bb7d1cf36fb15a98f; no voters:
I20260620 20:27:18.858604 20587 raft_consensus.cc:2781] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:18.859054 20587 raft_consensus.cc:686] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f [term 1 LEADER]: Becoming Leader. State: Replica: fdb3f7bbfd3b4e2bb7d1cf36fb15a98f, State: Running, Role: LEADER
I20260620 20:27:18.859573 20587 consensus_queue.cc:237] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3772f32d8a14f7882b2af2d4cc30be6" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41225 } } peers { permanent_uuid: "9e3d27ef13024e55a57d62de7cf93cd1" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35105 } } peers { permanent_uuid: "fdb3f7bbfd3b4e2bb7d1cf36fb15a98f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43917 } }
I20260620 20:27:18.864378 20115 catalog_manager.cc:5495] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f reported cstate change: term changed from 0 to 1, leader changed from <none> to fdb3f7bbfd3b4e2bb7d1cf36fb15a98f (127.31.235.130). New cstate: current_term: 1 leader_uuid: "fdb3f7bbfd3b4e2bb7d1cf36fb15a98f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3772f32d8a14f7882b2af2d4cc30be6" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e3d27ef13024e55a57d62de7cf93cd1" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdb3f7bbfd3b4e2bb7d1cf36fb15a98f" member_type: VOTER last_known_addr { host: "127.31.235.130" port: 43917 } health_report { overall_health: HEALTHY } } }
I20260620 20:27:18.898223 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:18.893158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58968 (local address 127.31.235.130:43917)
0620 20:27:18.893350 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:18.893355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:18.893379 (+ 24us) server_negotiation.cc:407] Connection header received
0620 20:27:18.893626 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:18.893630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:18.893718 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:18.893841 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:27:18.894155 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.894823 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.896113 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.896518 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.896607 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:18.897091 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:27:18.897614 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:27:18.897717 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:18.897833 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 20:27:18.897904 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 20:27:18.906410 20517 raft_consensus.cc:1261] T edf22a12b03045f29153b49d55351203 P 9e3d27ef13024e55a57d62de7cf93cd1 [term 1 FOLLOWER]: Refusing update from remote peer fdb3f7bbfd3b4e2bb7d1cf36fb15a98f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:27:18.906430 20256 raft_consensus.cc:1261] T edf22a12b03045f29153b49d55351203 P f3772f32d8a14f7882b2af2d4cc30be6 [term 1 FOLLOWER]: Refusing update from remote peer fdb3f7bbfd3b4e2bb7d1cf36fb15a98f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:27:18.907338 20587 consensus_queue.cc:1035] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3772f32d8a14f7882b2af2d4cc30be6" member_type: VOTER last_known_addr { host: "127.31.235.129" port: 41225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:27:18.907634 20591 consensus_queue.cc:1035] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3d27ef13024e55a57d62de7cf93cd1" member_type: VOTER last_known_addr { host: "127.31.235.131" port: 35105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:27:18.919575 20594 mvcc.cc:205] Tried to move back new op lower bound from 7299019730550730752 to 7299019730373013504. Current Snapshot: MvccSnapshot[applied={T|T < 7299019730550730752}]
I20260620 20:27:18.937665 20115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:27:18.937963 20115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 20:27:18.941867 20115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdde85a2cc8d4424abea062224c57c93: Sending DeleteTablet for 3 replicas of tablet edf22a12b03045f29153b49d55351203
I20260620 20:27:18.943096 20236 tablet_service.cc:1514] Processing DeleteTablet for tablet edf22a12b03045f29153b49d55351203 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:27:18 UTC) from {username='slave'} at 127.0.0.1:45234
I20260620 20:27:18.943203 20367 tablet_service.cc:1514] Processing DeleteTablet for tablet edf22a12b03045f29153b49d55351203 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:27:18 UTC) from {username='slave'} at 127.0.0.1:58958
I20260620 20:27:18.943188 20497 tablet_service.cc:1514] Processing DeleteTablet for tablet edf22a12b03045f29153b49d55351203 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 20:27:18 UTC) from {username='slave'} at 127.0.0.1:45292
I20260620 20:27:18.947031 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 20174
I20260620 20:27:18.955834 20604 tablet_replica.cc:330] T edf22a12b03045f29153b49d55351203 P f3772f32d8a14f7882b2af2d4cc30be6: stopping tablet replica
I20260620 20:27:18.957131 20604 raft_consensus.cc:2229] T edf22a12b03045f29153b49d55351203 P f3772f32d8a14f7882b2af2d4cc30be6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:27:18.958683 20605 tablet_replica.cc:330] T edf22a12b03045f29153b49d55351203 P 9e3d27ef13024e55a57d62de7cf93cd1: stopping tablet replica
I20260620 20:27:18.959347 20605 raft_consensus.cc:2229] T edf22a12b03045f29153b49d55351203 P 9e3d27ef13024e55a57d62de7cf93cd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 20:27:18.960043 20605 raft_consensus.cc:2258] T edf22a12b03045f29153b49d55351203 P 9e3d27ef13024e55a57d62de7cf93cd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:27:18.961834 20604 raft_consensus.cc:2258] T edf22a12b03045f29153b49d55351203 P f3772f32d8a14f7882b2af2d4cc30be6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:27:18.962209 20605 ts_tablet_manager.cc:1905] T edf22a12b03045f29153b49d55351203 P 9e3d27ef13024e55a57d62de7cf93cd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:27:18.964200 20604 ts_tablet_manager.cc:1905] T edf22a12b03045f29153b49d55351203 P f3772f32d8a14f7882b2af2d4cc30be6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:27:18.965716 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:18.948037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45236 (local address 127.31.235.129:41225)
0620 20:27:18.948277 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:18.948286 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:18.956377 (+ 8091us) server_negotiation.cc:407] Connection header received
0620 20:27:18.957445 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:18.957452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:18.957601 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:18.957692 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:18.958361 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.959528 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.961409 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:18.962085 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:18.962222 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:18.964055 (+ 1833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:18.964066 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:18.964083 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:18.964803 (+ 720us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:18.964878 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:18.964901 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:18.964985 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:18.965212 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 20:27:18.965332 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 20:27:18.966895 20296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:18.968575 20605 ts_tablet_manager.cc:1918] T edf22a12b03045f29153b49d55351203 P 9e3d27ef13024e55a57d62de7cf93cd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:27:18.968739 20605 log.cc:1196] T edf22a12b03045f29153b49d55351203 P 9e3d27ef13024e55a57d62de7cf93cd1: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-2/wal/wals/edf22a12b03045f29153b49d55351203
I20260620 20:27:18.969185 20605 ts_tablet_manager.cc:1939] T edf22a12b03045f29153b49d55351203 P 9e3d27ef13024e55a57d62de7cf93cd1: Deleting consensus metadata
I20260620 20:27:18.970860 20606 tablet_replica.cc:330] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f: stopping tablet replica
I20260620 20:27:18.970891 20102 catalog_manager.cc:4841] TS 9e3d27ef13024e55a57d62de7cf93cd1 (127.31.235.131:35105): tablet edf22a12b03045f29153b49d55351203 (table test-table [id=7aa21c66e2d7436282fe3c5c43deb1e7]) successfully deleted
I20260620 20:27:18.971436 20606 raft_consensus.cc:2229] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f [term 1 LEADER]: Raft consensus shutting down.
I20260620 20:27:18.972147 20606 raft_consensus.cc:2258] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 20:27:18.974718 20606 ts_tablet_manager.cc:1905] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 20:27:18.984009 20606 ts_tablet_manager.cc:1918] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 20:27:18.984202 20606 log.cc:1196] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f: Deleting WAL directory at /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781986916608810-32686-0/minicluster-data/ts-1/wal/wals/edf22a12b03045f29153b49d55351203
I20260620 20:27:18.984793 20606 ts_tablet_manager.cc:1939] T edf22a12b03045f29153b49d55351203 P fdb3f7bbfd3b4e2bb7d1cf36fb15a98f: Deleting consensus metadata
I20260620 20:27:18.986713 20102 catalog_manager.cc:4841] TS fdb3f7bbfd3b4e2bb7d1cf36fb15a98f (127.31.235.130:43917): tablet edf22a12b03045f29153b49d55351203 (table test-table [id=7aa21c66e2d7436282fe3c5c43deb1e7]) successfully deleted
I20260620 20:27:19.064180 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 20174
W20260620 20:27:19.079576 20099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.235.129:41225 (error 108)
I20260620 20:27:19.080297 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 20305
I20260620 20:27:19.081981 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:19.081388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:41225 (local address 127.0.0.1:45248)
0620 20:27:19.081652 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:19.081790 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 20:27:19.082252 20099 catalog_manager.cc:4584] TS f3772f32d8a14f7882b2af2d4cc30be6 (127.31.235.129:41225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet edf22a12b03045f29153b49d55351203: Network error: Client connection negotiation failed: client connection to 127.31.235.129:41225: connect: Connection refused (error 111)
I20260620 20:27:19.092244 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:19.081065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.31.235.130:43917)
0620 20:27:19.081237 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:19.081244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:19.081269 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:27:19.081482 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:19.081508 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:19.081621 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:19.081743 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:19.082234 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:19.083453 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:19.084751 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:19.085409 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:19.085595 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:19.090524 (+ 4929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:19.090536 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:19.090556 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:19.091378 (+ 822us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:19.091458 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:19.091484 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:19.091602 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:19.091738 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 20:27:19.091847 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 20:27:19.093051 20427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:19.135986 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:19.135500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:41225 (local address 127.0.0.1:45258)
0620 20:27:19.135722 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:19.135816 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 20:27:19.137148 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:19.136805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:41225 (local address 127.0.0.1:45274)
0620 20:27:19.136938 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:19.137006 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 20:27:19.197932 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:19.197394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:41225 (local address 127.0.0.1:45278)
0620 20:27:19.197688 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:19.197781 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260620 20:27:19.199033 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:19.198681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:41225 (local address 127.0.0.1:45292)
0620 20:27:19.198818 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:19.198902 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 20:27:19.207437 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 20305
I20260620 20:27:19.227016 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 20435
I20260620 20:27:19.234826 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:19.227793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45294 (local address 127.31.235.131:35105)
0620 20:27:19.227987 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:19.227994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:19.228013 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:27:19.228190 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:19.228201 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:19.228289 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:19.228407 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:19.228853 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:19.229727 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:19.231533 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:19.231915 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:19.232011 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:19.233192 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:19.233201 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:19.233216 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:19.234014 (+ 798us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:19.234072 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:19.234092 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:19.234185 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:19.234407 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 20:27:19.234494 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:27:19.235550 20557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:19.279556 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:19.279095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:41225 (local address 127.0.0.1:45300)
0620 20:27:19.279310 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:19.279404 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 20:27:19.280638 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:19.280253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:41225 (local address 127.0.0.1:45310)
0620 20:27:19.280455 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:19.280531 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 20:27:19.325184 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 20435
I20260620 20:27:19.345221 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 20084
I20260620 20:27:19.346454 20145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:19.457196 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:19.456629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.129:41225 (local address 127.0.0.1:45326)
0620 20:27:19.456873 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:19.456999 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.129:41225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 20:27:19.457198 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 20084
2026-06-20T20:27:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2755 ms)
[----------] 6 tests from EncryptionPolicyTest (22085 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:27:19 dist-test-slave-zzwf krb5kdc[20615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:27:19 dist-test-slave-zzwf krb5kdc[20615](info): set up 2 sockets
Jun 20 20:27:19 dist-test-slave-zzwf krb5kdc[20615](info): commencing operation
krb5kdc: starting...
W20260620 20:27:20.590374 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:27:20 dist-test-slave-zzwf krb5kdc[20615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:27:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:27:20.853407 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40503
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:21.194536 20628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:21.194837 20628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:21.195099 20628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:21.202639 20628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:21.202759 20628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:21.202868 20628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:21.202978 20628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:21.203058 20628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:21.213124 20628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40503
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:21.214583 20628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:21.215946 20628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:21.228564 20633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:21.228617 20636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:21.228467 20634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:21.229310 20628 server_base.cc:1034] running on GCE node
I20260620 20:27:21.230096 20628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:21.231516 20628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:21.232749 20628 hybrid_clock.cc:648] HybridClock initialized: now 1781987241232713 us; error 62 us; skew 500 ppm
I20260620 20:27:21.233098 20628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:21 dist-test-slave-zzwf krb5kdc[20615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:21.236334 20628 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:27:21.237844 20628 webserver.cc:460] Webserver started at http://127.0.0.1:42251/ using document root <none> and password file <none>
I20260620 20:27:21.238466 20628 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:21.238564 20628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:21.238821 20628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:21.241787 20628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "6f5cbcc0464a493fabc7c7c0be681a3d"
format_stamp: "Formatted at 2026-06-20 20:27:21 on dist-test-slave-zzwf"
server_key: "a482d6ef529fd64453a9855ce9bd8d6b"
server_key_iv: "875f6651e29bf51f312039475ad5b6c8"
server_key_version: "encryptionkey@0"
I20260620 20:27:21.242520 20628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "6f5cbcc0464a493fabc7c7c0be681a3d"
format_stamp: "Formatted at 2026-06-20 20:27:21 on dist-test-slave-zzwf"
server_key: "a482d6ef529fd64453a9855ce9bd8d6b"
server_key_iv: "875f6651e29bf51f312039475ad5b6c8"
server_key_version: "encryptionkey@0"
I20260620 20:27:21.247735 20628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260620 20:27:21.251281 20643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:21.252606 20628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:27:21.252789 20628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "6f5cbcc0464a493fabc7c7c0be681a3d"
format_stamp: "Formatted at 2026-06-20 20:27:21 on dist-test-slave-zzwf"
server_key: "a482d6ef529fd64453a9855ce9bd8d6b"
server_key_iv: "875f6651e29bf51f312039475ad5b6c8"
server_key_version: "encryptionkey@0"
I20260620 20:27:21.253000 20628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:21.271467 20628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:21.275954 20628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:21.276259 20628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:21.295519 20628 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40503
I20260620 20:27:21.295571 20694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40503 every 8 connection(s)
I20260620 20:27:21.296958 20628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:21.298261 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 20628
I20260620 20:27:21.298491 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:21.299315 32686 external_mini_cluster.cc:1428] Setting key 8ea8fcc578b5fc6e7983af76c397a741
I20260620 20:27:21.302011 20695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 20:27:21 dist-test-slave-zzwf krb5kdc[20615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 20:27:21.315047 20695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d: Bootstrap starting.
I20260620 20:27:21.318428 20695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:21.319554 20695 log.cc:824] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:21.322907 20695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d: No bootstrap required, opened a new log
I20260620 20:27:21.323797 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:21.301331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57631 (local address 127.0.0.1:40503)
0620 20:27:21.302154 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:21.302168 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:21.302225 (+ 57us) server_negotiation.cc:407] Connection header received
0620 20:27:21.304518 (+ 2293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:21.304544 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:21.305056 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:21.305531 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:21.306908 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:21.308973 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:21.310301 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:21.311325 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:21.311517 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:21.316006 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:21.316020 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:21.316038 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:21.316084 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:21.319319 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:21.320334 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:21.320342 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:21.320353 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:21.320573 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:21.321076 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:21.321083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:21.321085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:21.321651 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:21.321903 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:21.322423 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 20:27:21.322723 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":381,"threads_started":1}
I20260620 20:27:21.327643 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:27:21.332366 20695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5cbcc0464a493fabc7c7c0be681a3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40503 } }
I20260620 20:27:21.332818 20695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:21.332917 20695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f5cbcc0464a493fabc7c7c0be681a3d, State: Initialized, Role: FOLLOWER
I20260620 20:27:21.333727 20695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5cbcc0464a493fabc7c7c0be681a3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40503 } }
I20260620 20:27:21.334029 20695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:21.334177 20695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:21.334372 20695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:21.337275 20695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5cbcc0464a493fabc7c7c0be681a3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40503 } }
I20260620 20:27:21.337868 20695 leader_election.cc:302] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f5cbcc0464a493fabc7c7c0be681a3d; no voters:
I20260620 20:27:21.338513 20695 leader_election.cc:288] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:21.338629 20705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:21.339866 20705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d [term 1 LEADER]: Becoming Leader. State: Replica: 6f5cbcc0464a493fabc7c7c0be681a3d, State: Running, Role: LEADER
I20260620 20:27:21.340528 20705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5cbcc0464a493fabc7c7c0be681a3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40503 } }
I20260620 20:27:21.340691 20695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:21.343425 20707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f5cbcc0464a493fabc7c7c0be681a3d. Latest consensus state: current_term: 1 leader_uuid: "6f5cbcc0464a493fabc7c7c0be681a3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5cbcc0464a493fabc7c7c0be681a3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40503 } } }
I20260620 20:27:21.343889 20707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:21.343699 20706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f5cbcc0464a493fabc7c7c0be681a3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f5cbcc0464a493fabc7c7c0be681a3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40503 } } }
I20260620 20:27:21.344054 20706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:21.354435 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:21.333345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49823 (local address 127.0.0.1:40503)
0620 20:27:21.333484 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:21.333541 (+ 57us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:21.333972 (+ 431us) server_negotiation.cc:407] Connection header received
0620 20:27:21.334161 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:21.334170 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:21.334240 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:21.334349 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:21.336105 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:21.336899 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:21.338777 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:21.339687 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:21.339843 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:21.343312 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:21.343320 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:21.343324 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:21.343364 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:21.348907 (+ 5543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:21.350237 (+ 1330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:21.350245 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:21.350251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:21.350496 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:21.351214 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:21.351222 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:21.351228 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:21.351735 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:21.352102 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:21.353607 (+ 1505us) server_negotiation.cc:299] Negotiation successful
0620 20:27:21.353859 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 20:27:21.357667 20711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:21.364266 20711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:21.372907 20711 catalog_manager.cc:1347] Generated new cluster ID: aa106570ab1345e5ba40e3de6e069a63
I20260620 20:27:21.373070 20711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:21.387303 20711 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:21.389249 20711 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:21.397286 20711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f5cbcc0464a493fabc7c7c0be681a3d: Generated new TSK 0
I20260620 20:27:21.398339 20711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:21.406786 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 20628
I20260620 20:27:21.408803 20690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:21.499504 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 20628
W20260620 20:27:21.517611 20613 connection.cc:535] client connection to 127.0.0.1:40503 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40503): Connection reset by peer (error 104)
2026-06-20T20:27:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2033 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:27:21 dist-test-slave-zzwf krb5kdc[20727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:27:21 dist-test-slave-zzwf krb5kdc[20727](info): set up 2 sockets
Jun 20 20:27:21 dist-test-slave-zzwf krb5kdc[20727](info): commencing operation
krb5kdc: starting...
W20260620 20:27:22.634230 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:27:22 dist-test-slave-zzwf krb5kdc[20727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:27:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:27:22.893759 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:23.231168 20740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:23.231455 20740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:23.231612 20740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:23.239068 20740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:23.239171 20740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:23.239281 20740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:23.239398 20740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:23.239459 20740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:23.249225 20740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45057
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:23.250504 20740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:23.251770 20740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:23.263345 20745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:23.263614 20746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:23.263582 20748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:23.264631 20740 server_base.cc:1034] running on GCE node
I20260620 20:27:23.265444 20740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:23.266896 20740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:23.268091 20740 hybrid_clock.cc:648] HybridClock initialized: now 1781987243268015 us; error 71 us; skew 500 ppm
I20260620 20:27:23.268436 20740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:23 dist-test-slave-zzwf krb5kdc[20727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:23.271667 20740 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:27:23.273051 20740 webserver.cc:460] Webserver started at http://127.0.0.1:36383/ using document root <none> and password file <none>
I20260620 20:27:23.273775 20740 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:23.273880 20740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:23.274117 20740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:23.276968 20740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "1555c22ad4e049538b08490a23910962"
format_stamp: "Formatted at 2026-06-20 20:27:23 on dist-test-slave-zzwf"
server_key: "56d420da879121fc0b8b928b904a60e8"
server_key_iv: "fb16c66695584465234fd02e6b73ada0"
server_key_version: "encryptionkey@0"
I20260620 20:27:23.277745 20740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "1555c22ad4e049538b08490a23910962"
format_stamp: "Formatted at 2026-06-20 20:27:23 on dist-test-slave-zzwf"
server_key: "56d420da879121fc0b8b928b904a60e8"
server_key_iv: "fb16c66695584465234fd02e6b73ada0"
server_key_version: "encryptionkey@0"
I20260620 20:27:23.282778 20740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:27:23.286130 20755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:23.287685 20740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:27:23.287832 20740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "1555c22ad4e049538b08490a23910962"
format_stamp: "Formatted at 2026-06-20 20:27:23 on dist-test-slave-zzwf"
server_key: "56d420da879121fc0b8b928b904a60e8"
server_key_iv: "fb16c66695584465234fd02e6b73ada0"
server_key_version: "encryptionkey@0"
I20260620 20:27:23.287998 20740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:23.303944 20740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:23.308182 20740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:23.308431 20740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:23.328075 20740 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45057
I20260620 20:27:23.328104 20806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45057 every 8 connection(s)
I20260620 20:27:23.329654 20740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:23.333580 20807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:23.339335 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 20740
I20260620 20:27:23.339552 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:23.340003 32686 external_mini_cluster.cc:1428] Setting key 7cfe0af0adbb0bd621a1b8a1ba604ac2
I20260620 20:27:23.345443 20807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962: Bootstrap starting.
I20260620 20:27:23.349669 20807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:23.350744 20807 log.cc:824] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:23.353714 20807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962: No bootstrap required, opened a new log
I20260620 20:27:23.355633 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:23.342080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46703 (local address 127.0.0.1:45057)
0620 20:27:23.342985 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:23.343001 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:23.343057 (+ 56us) server_negotiation.cc:407] Connection header received
0620 20:27:23.345465 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:23.345520 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:23.346065 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:23.346536 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:23.347187 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:23.349226 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:23.350652 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:23.351680 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:23.351906 (+ 226us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:23.353179 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:23.353186 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:23.353200 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:23.353919 (+ 719us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:23.354008 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:23.354038 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:23.354216 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:23.354502 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 20:27:23.354669 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":394,"threads_started":1}
I20260620 20:27:23.359650 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:27:23.362600 20807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1555c22ad4e049538b08490a23910962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } }
I20260620 20:27:23.362994 20807 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:23.363116 20807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1555c22ad4e049538b08490a23910962, State: Initialized, Role: FOLLOWER
I20260620 20:27:23.363682 20807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1555c22ad4e049538b08490a23910962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } }
I20260620 20:27:23.363969 20807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:23.364087 20807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:23.364282 20807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:23.367205 20807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1555c22ad4e049538b08490a23910962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } }
I20260620 20:27:23.367806 20807 leader_election.cc:302] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1555c22ad4e049538b08490a23910962; no voters:
I20260620 20:27:23.368472 20807 leader_election.cc:288] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:23.368613 20817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:23.370565 20817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962 [term 1 LEADER]: Becoming Leader. State: Replica: 1555c22ad4e049538b08490a23910962, State: Running, Role: LEADER
I20260620 20:27:23.371193 20817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1555c22ad4e049538b08490a23910962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } }
I20260620 20:27:23.371429 20807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:23.371979 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:23.364167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36053 (local address 127.0.0.1:45057)
0620 20:27:23.364392 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:23.364400 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:23.364723 (+ 323us) server_negotiation.cc:407] Connection header received
0620 20:27:23.364931 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:23.364935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:23.365005 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:23.365087 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:23.365708 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:23.366580 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:23.368562 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:23.369509 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:23.369687 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:23.370941 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:23.370947 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:23.370950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:23.370981 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:23.371055 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:23.371101 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:23.371218 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:23.371564 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 20:27:23.371622 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 20:27:23.374600 20819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1555c22ad4e049538b08490a23910962. Latest consensus state: current_term: 1 leader_uuid: "1555c22ad4e049538b08490a23910962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1555c22ad4e049538b08490a23910962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } } }
I20260620 20:27:23.375130 20819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:23.375424 20818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1555c22ad4e049538b08490a23910962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1555c22ad4e049538b08490a23910962" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } } }
I20260620 20:27:23.375833 20818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:23.377643 20826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:23.383200 20826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:23.393422 20826 catalog_manager.cc:1347] Generated new cluster ID: 1f3b1f2ba8b94e31b65f3e89b636a216
I20260620 20:27:23.393610 20826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:23.407256 20826 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:23.409173 20826 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:23.421737 20826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1555c22ad4e049538b08490a23910962: Generated new TSK 0
I20260620 20:27:23.422741 20826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:23.434201 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 20740
W20260620 20:27:23.435675 20802 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46703
W20260620 20:27:23.436514 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 20740: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:27:23.436702 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 20740
W20260620 20:27:23.450717 20725 connection.cc:535] client connection to 127.0.0.1:45057 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45057): Connection reset by peer (error 104)
2026-06-20T20:27:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1933 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:27:23 dist-test-slave-zzwf krb5kdc[20838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:27:23 dist-test-slave-zzwf krb5kdc[20838](info): set up 2 sockets
Jun 20 20:27:23 dist-test-slave-zzwf krb5kdc[20838](info): commencing operation
krb5kdc: starting...
W20260620 20:27:24.546272 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:27:24 dist-test-slave-zzwf krb5kdc[20838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:27:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:27:24.813212 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33629
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:25.164552 20851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:25.164938 20851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:25.165122 20851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:25.172824 20851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:25.173019 20851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:25.173144 20851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:25.173272 20851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:25.173357 20851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:25.184031 20851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33629
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:25.185546 20851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:25.186913 20851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:25.199803 20856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:25.199908 20859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:25.199874 20857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:25.201009 20851 server_base.cc:1034] running on GCE node
I20260620 20:27:25.202025 20851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:25.203635 20851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:25.204885 20851 hybrid_clock.cc:648] HybridClock initialized: now 1781987245204830 us; error 46 us; skew 500 ppm
I20260620 20:27:25.205255 20851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:25 dist-test-slave-zzwf krb5kdc[20838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:25.212263 20851 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:27:25.214026 20851 webserver.cc:460] Webserver started at http://127.0.0.1:46251/ using document root <none> and password file <none>
I20260620 20:27:25.214763 20851 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:25.214920 20851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:25.215253 20851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:25.218518 20851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "09f7a5f1c95e483c81cb116409db1fbd"
format_stamp: "Formatted at 2026-06-20 20:27:25 on dist-test-slave-zzwf"
server_key: "9b30e54aadd3e4d62cf6bff0d703979c"
server_key_iv: "95b202243da8629910685df0de30050b"
server_key_version: "encryptionkey@0"
I20260620 20:27:25.219288 20851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "09f7a5f1c95e483c81cb116409db1fbd"
format_stamp: "Formatted at 2026-06-20 20:27:25 on dist-test-slave-zzwf"
server_key: "9b30e54aadd3e4d62cf6bff0d703979c"
server_key_iv: "95b202243da8629910685df0de30050b"
server_key_version: "encryptionkey@0"
I20260620 20:27:25.224521 20851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 20:27:25.228390 20866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:25.229873 20851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 20:27:25.230062 20851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "09f7a5f1c95e483c81cb116409db1fbd"
format_stamp: "Formatted at 2026-06-20 20:27:25 on dist-test-slave-zzwf"
server_key: "9b30e54aadd3e4d62cf6bff0d703979c"
server_key_iv: "95b202243da8629910685df0de30050b"
server_key_version: "encryptionkey@0"
I20260620 20:27:25.230265 20851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:25.235975 20851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:25.240325 20851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:25.240757 20851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:25.259956 20851 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33629
I20260620 20:27:25.260027 20917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33629 every 8 connection(s)
I20260620 20:27:25.261443 20851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:25.265306 20918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:25.268910 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 20851
I20260620 20:27:25.269083 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:25.269630 32686 external_mini_cluster.cc:1428] Setting key b11acf6087f9cefc06dc95dafd29bdb6
I20260620 20:27:25.278090 20918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd: Bootstrap starting.
I20260620 20:27:25.279877 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:25.271762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34737 (local address 127.0.0.1:33629)
0620 20:27:25.272771 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:25.272787 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:25.273185 (+ 398us) server_negotiation.cc:407] Connection header received
0620 20:27:25.276124 (+ 2939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:25.276155 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:25.276598 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:25.277017 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:25.278020 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:25.278028 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:25.278043 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:25.278060 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:25.278134 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:25.278163 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:25.278387 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:25.278507 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 20:27:25.278694 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":477,"threads_started":1}
I20260620 20:27:25.281833 20918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:25.283110 20918 log.cc:824] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:25.283938 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:27:25.286134 20918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd: No bootstrap required, opened a new log
I20260620 20:27:25.291124 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:25.288609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33209 (local address 127.0.0.1:33629)
0620 20:27:25.288782 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:25.288789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:25.289161 (+ 372us) server_negotiation.cc:407] Connection header received
0620 20:27:25.289347 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:25.289352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:25.289421 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:25.289569 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:25.290466 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:25.290472 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:25.290478 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:25.290501 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:25.290544 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:25.290567 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:25.290690 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:25.290752 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 20:27:25.290809 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 20:27:25.294955 20918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f7a5f1c95e483c81cb116409db1fbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33629 } }
I20260620 20:27:25.295303 20918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:25.295374 20918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09f7a5f1c95e483c81cb116409db1fbd, State: Initialized, Role: FOLLOWER
I20260620 20:27:25.295931 20918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f7a5f1c95e483c81cb116409db1fbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33629 } }
I20260620 20:27:25.296149 20918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:25.296245 20918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:25.296368 20918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:25.298672 20918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f7a5f1c95e483c81cb116409db1fbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33629 } }
I20260620 20:27:25.299160 20918 leader_election.cc:302] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09f7a5f1c95e483c81cb116409db1fbd; no voters:
I20260620 20:27:25.299747 20918 leader_election.cc:288] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:25.299902 20928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:25.300637 20928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd [term 1 LEADER]: Becoming Leader. State: Replica: 09f7a5f1c95e483c81cb116409db1fbd, State: Running, Role: LEADER
I20260620 20:27:25.301195 20928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f7a5f1c95e483c81cb116409db1fbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33629 } }
I20260620 20:27:25.302009 20918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:25.304067 20929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09f7a5f1c95e483c81cb116409db1fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f7a5f1c95e483c81cb116409db1fbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33629 } } }
I20260620 20:27:25.304390 20929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:25.306980 20930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09f7a5f1c95e483c81cb116409db1fbd. Latest consensus state: current_term: 1 leader_uuid: "09f7a5f1c95e483c81cb116409db1fbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09f7a5f1c95e483c81cb116409db1fbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33629 } } }
I20260620 20:27:25.307329 20930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:25.308302 20933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:25.314013 20933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:25.323745 20933 catalog_manager.cc:1347] Generated new cluster ID: c57538c227ae44c68635b75c7cbbc923
I20260620 20:27:25.323915 20933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:25.335577 20933 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:25.336002 20933 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:25.342502 20933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09f7a5f1c95e483c81cb116409db1fbd: Generated new TSK 0
I20260620 20:27:25.343355 20933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:25.353117 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 20851
W20260620 20:27:25.354250 20913 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34737
W20260620 20:27:25.355051 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 20851: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:27:25.355196 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 20851
2026-06-20T20:27:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1923 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:27:25 dist-test-slave-zzwf krb5kdc[20949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:27:25 dist-test-slave-zzwf krb5kdc[20949](info): set up 2 sockets
Jun 20 20:27:25 dist-test-slave-zzwf krb5kdc[20949](info): commencing operation
krb5kdc: starting...
W20260620 20:27:26.498405 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:27:26 dist-test-slave-zzwf krb5kdc[20949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:27:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:27:26.756650 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46187
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:27.089087 20962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:27.089378 20962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:27.089599 20962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:27.097105 20962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:27.097220 20962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:27.097340 20962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:27.097447 20962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:27.097554 20962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:27.107676 20962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:27.108978 20962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:27.110325 20962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:27.122157 20967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:27.122063 20970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:27.122100 20968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:27.122928 20962 server_base.cc:1034] running on GCE node
I20260620 20:27:27.123649 20962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:27.124997 20962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:27.126222 20962 hybrid_clock.cc:648] HybridClock initialized: now 1781987247126190 us; error 58 us; skew 500 ppm
I20260620 20:27:27.126631 20962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:27 dist-test-slave-zzwf krb5kdc[20949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:27.129812 20962 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:27:27.131387 20962 webserver.cc:460] Webserver started at http://127.0.0.1:41885/ using document root <none> and password file <none>
I20260620 20:27:27.132091 20962 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:27.132241 20962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:27.132553 20962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:27.135571 20962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "08d298d8035643799f1ac13c2f6b1f86"
format_stamp: "Formatted at 2026-06-20 20:27:27 on dist-test-slave-zzwf"
server_key: "75691fc000bc8bd20e16af49d73698fb"
server_key_iv: "de4723cbaa43885f7467bb9f17329161"
server_key_version: "encryptionkey@0"
I20260620 20:27:27.136344 20962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "08d298d8035643799f1ac13c2f6b1f86"
format_stamp: "Formatted at 2026-06-20 20:27:27 on dist-test-slave-zzwf"
server_key: "75691fc000bc8bd20e16af49d73698fb"
server_key_iv: "de4723cbaa43885f7467bb9f17329161"
server_key_version: "encryptionkey@0"
I20260620 20:27:27.141413 20962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 20:27:27.144836 20977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:27.146234 20962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 20:27:27.146461 20962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "08d298d8035643799f1ac13c2f6b1f86"
format_stamp: "Formatted at 2026-06-20 20:27:27 on dist-test-slave-zzwf"
server_key: "75691fc000bc8bd20e16af49d73698fb"
server_key_iv: "de4723cbaa43885f7467bb9f17329161"
server_key_version: "encryptionkey@0"
I20260620 20:27:27.146661 20962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:27.160493 20962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:27.164585 20962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:27.164891 20962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:27.183928 20962 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46187
I20260620 20:27:27.183981 21028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46187 every 8 connection(s)
I20260620 20:27:27.185379 20962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:27.189103 21029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:27.191112 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 20962
I20260620 20:27:27.191295 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:27.191841 32686 external_mini_cluster.cc:1428] Setting key 5f4335ea2a96a1f8243c8563fd1cb2d1
I20260620 20:27:27.204020 21029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86: Bootstrap starting.
Jun 20 20:27:27 dist-test-slave-zzwf krb5kdc[20949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 20:27:27.207803 21029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:27.208906 21029 log.cc:824] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:27.211905 21029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86: No bootstrap required, opened a new log
I20260620 20:27:27.214810 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:27.193713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36491 (local address 127.0.0.1:46187)
0620 20:27:27.194559 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:27.194569 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:27.194618 (+ 49us) server_negotiation.cc:407] Connection header received
0620 20:27:27.196225 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:27.196249 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:27.196671 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:27.196995 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:27.198795 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:27.200098 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:27.201953 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:27.202614 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:27.202763 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:27.207447 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:27.207455 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:27.207470 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:27.207515 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:27.210744 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:27.211502 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:27.211509 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:27.211514 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:27.211678 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:27.212126 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:27.212130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:27.212132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:27.212659 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:27.212906 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:27.213322 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 20:27:27.213763 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":398,"threads_started":1}
I20260620 20:27:27.218806 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:27:27.221774 21029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d298d8035643799f1ac13c2f6b1f86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46187 } }
I20260620 20:27:27.222283 21029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:27.222409 21029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d298d8035643799f1ac13c2f6b1f86, State: Initialized, Role: FOLLOWER
I20260620 20:27:27.223161 21029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d298d8035643799f1ac13c2f6b1f86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46187 } }
I20260620 20:27:27.223462 21029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:27.223614 21029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:27.223785 21029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:27.226624 21029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d298d8035643799f1ac13c2f6b1f86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46187 } }
I20260620 20:27:27.227264 21029 leader_election.cc:302] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08d298d8035643799f1ac13c2f6b1f86; no voters:
I20260620 20:27:27.227917 21029 leader_election.cc:288] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:27.228049 21039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:27.230055 21029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:27.230521 21039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86 [term 1 LEADER]: Becoming Leader. State: Replica: 08d298d8035643799f1ac13c2f6b1f86, State: Running, Role: LEADER
I20260620 20:27:27.231130 21039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d298d8035643799f1ac13c2f6b1f86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46187 } }
I20260620 20:27:27.234092 21041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08d298d8035643799f1ac13c2f6b1f86. Latest consensus state: current_term: 1 leader_uuid: "08d298d8035643799f1ac13c2f6b1f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d298d8035643799f1ac13c2f6b1f86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46187 } } }
I20260620 20:27:27.234483 21041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:27.237633 21040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08d298d8035643799f1ac13c2f6b1f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08d298d8035643799f1ac13c2f6b1f86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46187 } } }
I20260620 20:27:27.238711 21040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:27.244350 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:27.224348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45435 (local address 127.0.0.1:46187)
0620 20:27:27.224546 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:27.224557 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:27.224940 (+ 383us) server_negotiation.cc:407] Connection header received
0620 20:27:27.225075 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:27.225081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:27.225160 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:27.225283 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:27.226667 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:27.227735 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:27.229963 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:27.230907 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:27.231063 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:27.234397 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:27.234406 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:27.234415 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:27.234471 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:27.239209 (+ 4738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:27.240282 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:27.240291 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:27.240297 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:27.240505 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:27.241040 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:27.241048 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:27.241053 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:27.241622 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:27.241930 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:27.243508 (+ 1578us) server_negotiation.cc:299] Negotiation successful
0620 20:27:27.243800 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 20:27:27.246569 21047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:27.251919 21047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:27.261432 21047 catalog_manager.cc:1347] Generated new cluster ID: 3df2a4e6a86e4ba081fd04b290351150
I20260620 20:27:27.261596 21047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:27.291328 21047 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:27.293445 21047 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:27.313566 21047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08d298d8035643799f1ac13c2f6b1f86: Generated new TSK 0
I20260620 20:27:27.314713 21047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:27.331800 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 20962
I20260620 20:27:27.332864 21024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:27.428489 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 20962
2026-06-20T20:27:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2068 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:27:27 dist-test-slave-zzwf krb5kdc[21061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:27:27 dist-test-slave-zzwf krb5kdc[21061](info): set up 2 sockets
Jun 20 20:27:27 dist-test-slave-zzwf krb5kdc[21061](info): commencing operation
krb5kdc: starting...
W20260620 20:27:28.542553 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:27:28 dist-test-slave-zzwf krb5kdc[21061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:27:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:27:28.806515 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44625
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:29.141623 21074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:29.141968 21074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:29.142128 21074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:29.149228 21074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:29.149389 21074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:29.149518 21074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:29.149614 21074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:29.149704 21074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:29.159758 21074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:29.161100 21074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:29.162510 21074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:29.174327 21079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:29.174506 21082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:29.174360 21080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:29.175318 21074 server_base.cc:1034] running on GCE node
I20260620 20:27:29.176199 21074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:29.177733 21074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:29.178992 21074 hybrid_clock.cc:648] HybridClock initialized: now 1781987249178926 us; error 54 us; skew 500 ppm
I20260620 20:27:29.179391 21074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:29 dist-test-slave-zzwf krb5kdc[21061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:29.182814 21074 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:27:29.184337 21074 webserver.cc:460] Webserver started at http://127.0.0.1:33253/ using document root <none> and password file <none>
I20260620 20:27:29.185056 21074 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:29.185217 21074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:29.185601 21074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:29.188902 21074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "861d63cd75a94b7a9e4f167a0744f929"
format_stamp: "Formatted at 2026-06-20 20:27:29 on dist-test-slave-zzwf"
server_key: "f5c1044bef2c0fc9b4db71f36b174d92"
server_key_iv: "73519967f1d26189dd4912114e3aa84b"
server_key_version: "encryptionkey@0"
I20260620 20:27:29.189836 21074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "861d63cd75a94b7a9e4f167a0744f929"
format_stamp: "Formatted at 2026-06-20 20:27:29 on dist-test-slave-zzwf"
server_key: "f5c1044bef2c0fc9b4db71f36b174d92"
server_key_iv: "73519967f1d26189dd4912114e3aa84b"
server_key_version: "encryptionkey@0"
I20260620 20:27:29.195680 21074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:27:29.199273 21089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:29.200667 21074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:27:29.200865 21074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "861d63cd75a94b7a9e4f167a0744f929"
format_stamp: "Formatted at 2026-06-20 20:27:29 on dist-test-slave-zzwf"
server_key: "f5c1044bef2c0fc9b4db71f36b174d92"
server_key_iv: "73519967f1d26189dd4912114e3aa84b"
server_key_version: "encryptionkey@0"
I20260620 20:27:29.201076 21074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:29.219030 21074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:29.223156 21074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:29.223465 21074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:29.242089 21074 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44625
I20260620 20:27:29.242120 21140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44625 every 8 connection(s)
I20260620 20:27:29.243532 21074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:29.247216 21141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:29.253469 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 21074
I20260620 20:27:29.253742 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:29.254199 32686 external_mini_cluster.cc:1428] Setting key dfeb2e61c50625e39ef15bd9413d67b8
I20260620 20:27:29.257922 21141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929: Bootstrap starting.
I20260620 20:27:29.261229 21141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:29.262599 21141 log.cc:824] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:29.265480 21141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929: No bootstrap required, opened a new log
I20260620 20:27:29.267792 21144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:29.256016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43967 (local address 127.0.0.1:44625)
0620 20:27:29.256827 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:29.256839 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:29.256888 (+ 49us) server_negotiation.cc:407] Connection header received
0620 20:27:29.258711 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:29.258734 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:29.259123 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:29.259480 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:29.260091 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:29.261570 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:29.263479 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:29.264145 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:29.264348 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:29.265547 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:29.265563 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:29.265579 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:29.266151 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:29.266228 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:29.266252 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:29.266378 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:29.266611 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 20:27:29.266854 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":406,"threads_started":1}
I20260620 20:27:29.271754 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:27:29.274269 21141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861d63cd75a94b7a9e4f167a0744f929" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44625 } }
I20260620 20:27:29.274665 21141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:29.274806 21141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861d63cd75a94b7a9e4f167a0744f929, State: Initialized, Role: FOLLOWER
I20260620 20:27:29.275450 21141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861d63cd75a94b7a9e4f167a0744f929" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44625 } }
I20260620 20:27:29.275689 21141 raft_consensus.cc:388] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:29.275837 21141 raft_consensus.cc:482] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:29.276065 21141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:29.278446 21141 raft_consensus.cc:504] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861d63cd75a94b7a9e4f167a0744f929" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44625 } }
I20260620 20:27:29.278949 21141 leader_election.cc:302] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 861d63cd75a94b7a9e4f167a0744f929; no voters:
I20260620 20:27:29.279603 21141 leader_election.cc:288] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:29.279723 21151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:29.280827 21151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929 [term 1 LEADER]: Becoming Leader. State: Replica: 861d63cd75a94b7a9e4f167a0744f929, State: Running, Role: LEADER
I20260620 20:27:29.281538 21151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861d63cd75a94b7a9e4f167a0744f929" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44625 } }
I20260620 20:27:29.281919 21141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:29.284292 21144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:29.276139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52995 (local address 127.0.0.1:44625)
0620 20:27:29.276400 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:29.276410 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:29.276942 (+ 532us) server_negotiation.cc:407] Connection header received
0620 20:27:29.277158 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:29.277163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:29.277240 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:29.277354 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:29.277874 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:29.278633 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:29.280465 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:29.281274 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:29.281386 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:29.282889 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:29.282896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:29.282903 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:29.282928 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:29.282974 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:29.282997 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:29.283101 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:29.283656 (+ 555us) server_negotiation.cc:299] Negotiation successful
0620 20:27:29.283795 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 20:27:29.284816 21152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "861d63cd75a94b7a9e4f167a0744f929" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861d63cd75a94b7a9e4f167a0744f929" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44625 } } }
I20260620 20:27:29.285286 21152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:29.285586 21153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 861d63cd75a94b7a9e4f167a0744f929. Latest consensus state: current_term: 1 leader_uuid: "861d63cd75a94b7a9e4f167a0744f929" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861d63cd75a94b7a9e4f167a0744f929" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44625 } } }
I20260620 20:27:29.287186 21153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:29.288110 21156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:29.293536 21156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:29.302702 21156 catalog_manager.cc:1347] Generated new cluster ID: a46f689946d241bb91b1028d70a80aec
I20260620 20:27:29.302831 21156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:29.315982 21156 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:29.317335 21156 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:29.328694 21156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 861d63cd75a94b7a9e4f167a0744f929: Generated new TSK 0
I20260620 20:27:29.329811 21156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:29.343483 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 21074
W20260620 20:27:29.344660 21136 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43967
W20260620 20:27:29.345479 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 21074: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:27:29.345654 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 21074
2026-06-20T20:27:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1921 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:27:29 dist-test-slave-zzwf krb5kdc[21172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:27:29 dist-test-slave-zzwf krb5kdc[21172](info): set up 2 sockets
Jun 20 20:27:29 dist-test-slave-zzwf krb5kdc[21172](info): commencing operation
krb5kdc: starting...
W20260620 20:27:30.478271 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:27:30 dist-test-slave-zzwf krb5kdc[21172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:27:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:27:30.744412 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:31.074636 21185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:31.074952 21185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:31.075143 21185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:31.082530 21185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:31.082644 21185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:31.082742 21185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:31.082846 21185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:31.082935 21185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:31.092926 21185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:31.094286 21185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:31.095638 21185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:31.107880 21191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:31.108000 21193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:31.108016 21190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:31.109133 21185 server_base.cc:1034] running on GCE node
I20260620 20:27:31.110008 21185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:31.111436 21185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:31.112672 21185 hybrid_clock.cc:648] HybridClock initialized: now 1781987251112629 us; error 37 us; skew 500 ppm
I20260620 20:27:31.113034 21185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:31 dist-test-slave-zzwf krb5kdc[21172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:31.121232 21185 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:27:31.122946 21185 webserver.cc:460] Webserver started at http://127.0.0.1:43251/ using document root <none> and password file <none>
I20260620 20:27:31.123659 21185 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:31.123807 21185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:31.124119 21185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:31.127190 21185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "7660897b5be54295a10c6d1c3106d708"
format_stamp: "Formatted at 2026-06-20 20:27:31 on dist-test-slave-zzwf"
server_key: "6365d642be5f5a0280dedb1dc52f9c5a"
server_key_iv: "cefa90a3f74b3200d916655e7d187864"
server_key_version: "encryptionkey@0"
I20260620 20:27:31.127974 21185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "7660897b5be54295a10c6d1c3106d708"
format_stamp: "Formatted at 2026-06-20 20:27:31 on dist-test-slave-zzwf"
server_key: "6365d642be5f5a0280dedb1dc52f9c5a"
server_key_iv: "cefa90a3f74b3200d916655e7d187864"
server_key_version: "encryptionkey@0"
I20260620 20:27:31.133314 21185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:27:31.136905 21200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:31.138375 21185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:27:31.138557 21185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "7660897b5be54295a10c6d1c3106d708"
format_stamp: "Formatted at 2026-06-20 20:27:31 on dist-test-slave-zzwf"
server_key: "6365d642be5f5a0280dedb1dc52f9c5a"
server_key_iv: "cefa90a3f74b3200d916655e7d187864"
server_key_version: "encryptionkey@0"
I20260620 20:27:31.138759 21185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:31.144575 21185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:31.148957 21185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:31.149252 21185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:31.168290 21185 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44475
I20260620 20:27:31.168385 21251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44475 every 8 connection(s)
I20260620 20:27:31.169682 21185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:31.173326 21252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:31.179198 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 21185
I20260620 20:27:31.179404 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:31.179908 32686 external_mini_cluster.cc:1428] Setting key 494ffc6894757028aaf4f137ef05b670
I20260620 20:27:31.184350 21252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708: Bootstrap starting.
I20260620 20:27:31.187707 21255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:31.181728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58279 (local address 127.0.0.1:44475)
0620 20:27:31.182606 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:31.182618 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:31.182663 (+ 45us) server_negotiation.cc:407] Connection header received
0620 20:27:31.184361 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:31.184381 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:31.184763 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:31.185075 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:31.186204 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:31.186211 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:31.186223 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:31.186240 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:31.186299 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:31.186328 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:31.186499 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:31.186606 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 20:27:31.186814 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":417,"threads_started":1}
I20260620 20:27:31.187978 21252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:31.189023 21252 log.cc:824] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:31.191510 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:27:31.192075 21252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708: No bootstrap required, opened a new log
I20260620 20:27:31.198328 21255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:31.195901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53457 (local address 127.0.0.1:44475)
0620 20:27:31.196111 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:31.196117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:31.196450 (+ 333us) server_negotiation.cc:407] Connection header received
0620 20:27:31.196585 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:31.196590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:31.196653 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:31.196756 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:31.197683 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:31.197688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:31.197693 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:31.197711 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:31.197767 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:31.197786 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:31.197906 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:31.197981 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 20:27:31.198037 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:27:31.200683 21252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7660897b5be54295a10c6d1c3106d708" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } }
I20260620 20:27:31.201028 21252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:31.201112 21252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7660897b5be54295a10c6d1c3106d708, State: Initialized, Role: FOLLOWER
I20260620 20:27:31.201748 21252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7660897b5be54295a10c6d1c3106d708" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } }
I20260620 20:27:31.201968 21252 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:31.202065 21252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:31.202193 21252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:31.204310 21252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7660897b5be54295a10c6d1c3106d708" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } }
I20260620 20:27:31.204737 21252 leader_election.cc:302] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7660897b5be54295a10c6d1c3106d708; no voters:
I20260620 20:27:31.205314 21252 leader_election.cc:288] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:31.205474 21262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:31.206430 21262 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708 [term 1 LEADER]: Becoming Leader. State: Replica: 7660897b5be54295a10c6d1c3106d708, State: Running, Role: LEADER
I20260620 20:27:31.207028 21262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7660897b5be54295a10c6d1c3106d708" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } }
I20260620 20:27:31.207839 21252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:31.210223 21263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7660897b5be54295a10c6d1c3106d708" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7660897b5be54295a10c6d1c3106d708" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } } }
I20260620 20:27:31.210235 21264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7660897b5be54295a10c6d1c3106d708. Latest consensus state: current_term: 1 leader_uuid: "7660897b5be54295a10c6d1c3106d708" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7660897b5be54295a10c6d1c3106d708" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } } }
I20260620 20:27:31.210670 21263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:31.210685 21264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:31.211664 21269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:31.219797 21269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:31.231905 21269 catalog_manager.cc:1347] Generated new cluster ID: a53d00560b704c779c29aeb64b72556c
I20260620 20:27:31.232133 21269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:31.256836 21269 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:31.257366 21269 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:31.267166 21269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7660897b5be54295a10c6d1c3106d708: Generated new TSK 0
I20260620 20:27:31.268214 21269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:31.283087 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 21185
W20260620 20:27:31.284173 21247 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58279
W20260620 20:27:31.284869 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 21185: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:27:31.285012 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 21185
2026-06-20T20:27:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1937 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:27:31 dist-test-slave-zzwf krb5kdc[21283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:27:31 dist-test-slave-zzwf krb5kdc[21283](info): set up 2 sockets
Jun 20 20:27:31 dist-test-slave-zzwf krb5kdc[21283](info): commencing operation
krb5kdc: starting...
W20260620 20:27:32.398651 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:27:32 dist-test-slave-zzwf krb5kdc[21283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:27:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:27:32.675222 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:33.008736 21296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:33.009052 21296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:33.009218 21296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:33.016916 21296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:33.017047 21296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:33.017163 21296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:33.017295 21296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:33.017393 21296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:33.027837 21296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:33.029373 21296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:33.030750 21296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:33.043036 21301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:33.043036 21302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:33.043197 21304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:33.044158 21296 server_base.cc:1034] running on GCE node
I20260620 20:27:33.044907 21296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:33.046381 21296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:33.047657 21296 hybrid_clock.cc:648] HybridClock initialized: now 1781987253047595 us; error 35 us; skew 500 ppm
I20260620 20:27:33.048058 21296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:33 dist-test-slave-zzwf krb5kdc[21283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:33.051317 21296 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:27:33.052829 21296 webserver.cc:460] Webserver started at http://127.0.0.1:32909/ using document root <none> and password file <none>
I20260620 20:27:33.053469 21296 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:33.053637 21296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:33.053905 21296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:33.057035 21296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "4945816a4ebd494c96f2fb5ef7b3a459"
format_stamp: "Formatted at 2026-06-20 20:27:33 on dist-test-slave-zzwf"
server_key: "17017c5a3a8e027b86965444189ded2a"
server_key_iv: "1061339c78a4ef387010b23696b4ce39"
server_key_version: "encryptionkey@0"
I20260620 20:27:33.057839 21296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "4945816a4ebd494c96f2fb5ef7b3a459"
format_stamp: "Formatted at 2026-06-20 20:27:33 on dist-test-slave-zzwf"
server_key: "17017c5a3a8e027b86965444189ded2a"
server_key_iv: "1061339c78a4ef387010b23696b4ce39"
server_key_version: "encryptionkey@0"
I20260620 20:27:33.062912 21296 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 20:27:33.066496 21311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:33.067842 21296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:27:33.068044 21296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "4945816a4ebd494c96f2fb5ef7b3a459"
format_stamp: "Formatted at 2026-06-20 20:27:33 on dist-test-slave-zzwf"
server_key: "17017c5a3a8e027b86965444189ded2a"
server_key_iv: "1061339c78a4ef387010b23696b4ce39"
server_key_version: "encryptionkey@0"
I20260620 20:27:33.068239 21296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:33.083506 21296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:33.088068 21296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:33.088397 21296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:33.111073 21296 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38799
I20260620 20:27:33.111097 21362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38799 every 8 connection(s)
I20260620 20:27:33.112804 21296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:33.116986 21363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:33.121204 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 21296
I20260620 20:27:33.121433 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:33.122125 32686 external_mini_cluster.cc:1428] Setting key 3d2b567010a42851acbc7e6e32b7c700
I20260620 20:27:33.131273 21363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459: Bootstrap starting.
I20260620 20:27:33.135262 21363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459: Neither blocks nor log segments found. Creating new log.
Jun 20 20:27:33 dist-test-slave-zzwf krb5kdc[21283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 20:27:33.136564 21363 log.cc:824] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:33.140717 21363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459: No bootstrap required, opened a new log
I20260620 20:27:33.148399 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:33.124361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:35721 (local address 127.0.0.1:38799)
0620 20:27:33.125189 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:33.125201 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:33.125251 (+ 50us) server_negotiation.cc:407] Connection header received
0620 20:27:33.127046 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:33.127070 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:33.127523 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:33.127941 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:33.129456 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:33.130826 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:33.132198 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:33.132908 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:33.133123 (+ 215us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:33.137748 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:33.137759 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:33.137783 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:33.137831 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:33.142586 (+ 4755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:33.143613 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:33.143623 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:33.143634 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:33.143914 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:33.144500 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:33.144507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:33.144511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:33.145396 (+ 885us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:33.145827 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:33.146307 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 20:27:33.146716 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":379,"threads_started":1}
I20260620 20:27:33.153038 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:27:33.153439 21363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4945816a4ebd494c96f2fb5ef7b3a459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } }
I20260620 20:27:33.153972 21363 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:33.154101 21363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4945816a4ebd494c96f2fb5ef7b3a459, State: Initialized, Role: FOLLOWER
I20260620 20:27:33.154819 21363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4945816a4ebd494c96f2fb5ef7b3a459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } }
I20260620 20:27:33.155138 21363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:33.155325 21363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:33.155525 21363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:33.158782 21363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4945816a4ebd494c96f2fb5ef7b3a459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } }
I20260620 20:27:33.159463 21363 leader_election.cc:302] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4945816a4ebd494c96f2fb5ef7b3a459; no voters:
I20260620 20:27:33.160319 21363 leader_election.cc:288] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:33.160439 21373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:33.163641 21363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:33.163828 21373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459 [term 1 LEADER]: Becoming Leader. State: Replica: 4945816a4ebd494c96f2fb5ef7b3a459, State: Running, Role: LEADER
I20260620 20:27:33.164677 21373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4945816a4ebd494c96f2fb5ef7b3a459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } }
I20260620 20:27:33.169256 21374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4945816a4ebd494c96f2fb5ef7b3a459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4945816a4ebd494c96f2fb5ef7b3a459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } } }
I20260620 20:27:33.169878 21374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:33.169756 21375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4945816a4ebd494c96f2fb5ef7b3a459. Latest consensus state: current_term: 1 leader_uuid: "4945816a4ebd494c96f2fb5ef7b3a459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4945816a4ebd494c96f2fb5ef7b3a459" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38799 } } }
I20260620 20:27:33.170147 21375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:33.182524 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:33.157894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:46169 (local address 127.0.0.1:38799)
0620 20:27:33.158101 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:33.158109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:33.158424 (+ 315us) server_negotiation.cc:407] Connection header received
0620 20:27:33.158573 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:33.158582 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:33.158698 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:33.158852 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:33.160984 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:33.162094 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:33.163554 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:33.164303 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:33.164488 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:33.168850 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:33.168867 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:33.168875 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:33.168967 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:33.175419 (+ 6452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:33.177255 (+ 1836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:33.177266 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:33.177274 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:33.177532 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:33.178515 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:33.178525 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:33.178532 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:33.179255 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:33.179661 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:33.181425 (+ 1764us) server_negotiation.cc:299] Negotiation successful
0620 20:27:33.181766 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 20:27:33.183653 21381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:33.191067 21381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:33.201378 21381 catalog_manager.cc:1347] Generated new cluster ID: 99ca0040ab834a99aa370f8e8c21f7b7
I20260620 20:27:33.201561 21381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:33.226503 21381 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:33.228739 21381 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:33.244053 21381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4945816a4ebd494c96f2fb5ef7b3a459: Generated new TSK 0
I20260620 20:27:33.245221 21381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:33.251709 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 21296
I20260620 20:27:33.253649 21358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:33.344008 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 21296
W20260620 20:27:33.361685 21281 connection.cc:535] client connection to 127.0.0.1:38799 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38799): Connection reset by peer (error 104)
2026-06-20T20:27:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2059 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:27:33 dist-test-slave-zzwf krb5kdc[21395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:27:33 dist-test-slave-zzwf krb5kdc[21395](info): set up 2 sockets
Jun 20 20:27:33 dist-test-slave-zzwf krb5kdc[21395](info): commencing operation
krb5kdc: starting...
W20260620 20:27:34.462383 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:27:34 dist-test-slave-zzwf krb5kdc[21395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:27:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:27:34.732524 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37505
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:35.069723 21408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:35.070078 21408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:35.070238 21408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:35.077711 21408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:35.077878 21408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:35.077984 21408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:35.078096 21408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:35.078177 21408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:35.088276 21408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:35.089664 21408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:35.090971 21408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:35.103308 21413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:35.103276 21416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:35.103226 21414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:35.104398 21408 server_base.cc:1034] running on GCE node
I20260620 20:27:35.105209 21408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:35.106747 21408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:35.108007 21408 hybrid_clock.cc:648] HybridClock initialized: now 1781987255107957 us; error 55 us; skew 500 ppm
I20260620 20:27:35.108424 21408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:35 dist-test-slave-zzwf krb5kdc[21395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:35.111735 21408 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:27:35.113425 21408 webserver.cc:460] Webserver started at http://127.0.0.1:42231/ using document root <none> and password file <none>
I20260620 20:27:35.114143 21408 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:35.114303 21408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:35.114609 21408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:35.117676 21408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "c8993b0ca27141adac97180038c50e47"
format_stamp: "Formatted at 2026-06-20 20:27:35 on dist-test-slave-zzwf"
server_key: "e2e208b128ca0b981b26e4e7e50e06cd"
server_key_iv: "4fd0f3a4b55017931e469a3319fa0513"
server_key_version: "encryptionkey@0"
I20260620 20:27:35.118463 21408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "c8993b0ca27141adac97180038c50e47"
format_stamp: "Formatted at 2026-06-20 20:27:35 on dist-test-slave-zzwf"
server_key: "e2e208b128ca0b981b26e4e7e50e06cd"
server_key_iv: "4fd0f3a4b55017931e469a3319fa0513"
server_key_version: "encryptionkey@0"
I20260620 20:27:35.123678 21408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:27:35.127360 21423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:35.128727 21408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:27:35.128939 21408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "c8993b0ca27141adac97180038c50e47"
format_stamp: "Formatted at 2026-06-20 20:27:35 on dist-test-slave-zzwf"
server_key: "e2e208b128ca0b981b26e4e7e50e06cd"
server_key_iv: "4fd0f3a4b55017931e469a3319fa0513"
server_key_version: "encryptionkey@0"
I20260620 20:27:35.129149 21408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:35.145671 21408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:35.149916 21408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:35.150235 21408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:35.169561 21408 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37505
I20260620 20:27:35.169646 21474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37505 every 8 connection(s)
I20260620 20:27:35.170984 21408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:35.174868 21475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:35.177950 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 21408
I20260620 20:27:35.178123 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:35.178681 32686 external_mini_cluster.cc:1428] Setting key c8c8229b02e021b2310ccecdcf242ce7
I20260620 20:27:35.187234 21475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47: Bootstrap starting.
I20260620 20:27:35.190542 21475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:35.191661 21475 log.cc:824] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:35.194460 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:35.180601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:35773 (local address 127.0.0.1:37505)
0620 20:27:35.181477 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:35.181514 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:35.181559 (+ 45us) server_negotiation.cc:407] Connection header received
0620 20:27:35.183255 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:35.183277 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:35.183916 (+ 639us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:35.184362 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:35.185131 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:35.186663 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:35.188323 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:35.189160 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:35.189428 (+ 268us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:35.191083 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:35.191092 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:35.191108 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:35.191743 (+ 635us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:35.191889 (+ 146us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:35.191927 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:35.192147 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:35.192869 (+ 722us) server_negotiation.cc:299] Negotiation successful
0620 20:27:35.193107 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":360,"threads_started":1}
I20260620 20:27:35.195218 21475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47: No bootstrap required, opened a new log
I20260620 20:27:35.198697 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:27:35.203744 21475 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8993b0ca27141adac97180038c50e47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37505 } }
I20260620 20:27:35.204319 21475 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:35.204473 21475 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8993b0ca27141adac97180038c50e47, State: Initialized, Role: FOLLOWER
I20260620 20:27:35.205135 21475 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8993b0ca27141adac97180038c50e47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37505 } }
I20260620 20:27:35.205416 21475 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:35.205636 21475 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:35.205832 21475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:35.208968 21475 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8993b0ca27141adac97180038c50e47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37505 } }
I20260620 20:27:35.209656 21475 leader_election.cc:302] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8993b0ca27141adac97180038c50e47; no voters:
I20260620 20:27:35.210404 21475 leader_election.cc:288] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:35.210537 21485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:35.210889 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:35.203154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:50155 (local address 127.0.0.1:37505)
0620 20:27:35.203433 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:35.203440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:35.203829 (+ 389us) server_negotiation.cc:407] Connection header received
0620 20:27:35.204030 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:35.204034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:35.204102 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:35.204230 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:35.204847 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:35.206096 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:35.207978 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:35.208467 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:35.208571 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:35.209909 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:35.209916 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:35.209921 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:35.209941 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:35.210004 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:35.210022 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:35.210108 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:35.210417 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 20:27:35.210478 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:27:35.212128 21485 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47 [term 1 LEADER]: Becoming Leader. State: Replica: c8993b0ca27141adac97180038c50e47, State: Running, Role: LEADER
I20260620 20:27:35.212704 21485 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8993b0ca27141adac97180038c50e47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37505 } }
I20260620 20:27:35.213129 21475 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:35.215422 21486 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8993b0ca27141adac97180038c50e47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8993b0ca27141adac97180038c50e47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37505 } } }
I20260620 20:27:35.215816 21486 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:35.216408 21487 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8993b0ca27141adac97180038c50e47. Latest consensus state: current_term: 1 leader_uuid: "c8993b0ca27141adac97180038c50e47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8993b0ca27141adac97180038c50e47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37505 } } }
I20260620 20:27:35.216698 21487 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:35.218117 21491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:35.223460 21491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:35.233887 21491 catalog_manager.cc:1347] Generated new cluster ID: 1e6480230e3648d984beafe48d7b46fb
I20260620 20:27:35.234018 21491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:35.257222 21491 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:35.258612 21491 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:35.263653 21491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8993b0ca27141adac97180038c50e47: Generated new TSK 0
I20260620 20:27:35.264473 21491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:35.269979 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 21408
W20260620 20:27:35.271323 21470 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.242:35773
W20260620 20:27:35.272089 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 21408: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:27:35.272275 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 21408
W20260620 20:27:35.289445 21393 connection.cc:535] client connection to 127.0.0.1:37505 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37505): Connection reset by peer (error 104)
2026-06-20T20:27:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1927 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:27:35 dist-test-slave-zzwf krb5kdc[21507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:27:35 dist-test-slave-zzwf krb5kdc[21507](info): set up 2 sockets
Jun 20 20:27:35 dist-test-slave-zzwf krb5kdc[21507](info): commencing operation
krb5kdc: starting...
W20260620 20:27:36.382421 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:27:36 dist-test-slave-zzwf krb5kdc[21507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:27:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:27:36.645843 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37807
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:36.981712 21520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:36.982013 21520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:36.982105 21520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:36.989284 21520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:36.989409 21520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:36.989457 21520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:36.989574 21520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:36.989650 21520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:36.999620 21520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:37.000960 21520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:37.002236 21520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:37.014245 21525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:37.014333 21528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:37.014420 21526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:37.015074 21520 server_base.cc:1034] running on GCE node
I20260620 20:27:37.015863 21520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:37.017351 21520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:37.018587 21520 hybrid_clock.cc:648] HybridClock initialized: now 1781987257018539 us; error 46 us; skew 500 ppm
I20260620 20:27:37.018985 21520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:37 dist-test-slave-zzwf krb5kdc[21507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:37.026465 21520 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:27:37.028098 21520 webserver.cc:460] Webserver started at http://127.0.0.1:37799/ using document root <none> and password file <none>
I20260620 20:27:37.028748 21520 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:37.028851 21520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:37.029110 21520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:37.032066 21520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "1117d7342a3b443bb371ea4ad77b8165"
format_stamp: "Formatted at 2026-06-20 20:27:37 on dist-test-slave-zzwf"
server_key: "73de026c14344268464e4bd08760b780"
server_key_iv: "90cfa8b47b21db236c50c21df9a3a31f"
server_key_version: "encryptionkey@0"
I20260620 20:27:37.032938 21520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "1117d7342a3b443bb371ea4ad77b8165"
format_stamp: "Formatted at 2026-06-20 20:27:37 on dist-test-slave-zzwf"
server_key: "73de026c14344268464e4bd08760b780"
server_key_iv: "90cfa8b47b21db236c50c21df9a3a31f"
server_key_version: "encryptionkey@0"
I20260620 20:27:37.038184 21520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 20:27:37.041993 21535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:37.043391 21520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:27:37.043593 21520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "1117d7342a3b443bb371ea4ad77b8165"
format_stamp: "Formatted at 2026-06-20 20:27:37 on dist-test-slave-zzwf"
server_key: "73de026c14344268464e4bd08760b780"
server_key_iv: "90cfa8b47b21db236c50c21df9a3a31f"
server_key_version: "encryptionkey@0"
I20260620 20:27:37.043803 21520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:37.050177 21520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:37.055230 21520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:37.055565 21520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:37.075817 21520 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37807
I20260620 20:27:37.075920 21586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37807 every 8 connection(s)
I20260620 20:27:37.077365 21520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:37.081310 21587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:37.081526 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 21520
I20260620 20:27:37.081700 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:37.082234 32686 external_mini_cluster.cc:1428] Setting key 59f428463e1e68426c6461faad4a9daa
I20260620 20:27:37.090978 21590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:37.084115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:33539 (local address 127.0.0.1:37807)
0620 20:27:37.085060 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:37.085072 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:37.085117 (+ 45us) server_negotiation.cc:407] Connection header received
0620 20:27:37.087515 (+ 2398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:37.087543 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:37.087902 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:37.088259 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:37.089375 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:37.089382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:37.089394 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:37.089409 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:37.089471 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:37.089553 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:37.089742 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:37.089848 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 20:27:37.090053 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":463,"threads_started":1}
I20260620 20:27:37.094949 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:27:37.095178 21587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165: Bootstrap starting.
I20260620 20:27:37.098198 21587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:37.099511 21587 log.cc:824] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:37.102509 21587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165: No bootstrap required, opened a new log
I20260620 20:27:37.102550 21590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:37.099597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:45965 (local address 127.0.0.1:37807)
0620 20:27:37.099837 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:37.099845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:37.100108 (+ 263us) server_negotiation.cc:407] Connection header received
0620 20:27:37.100391 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:37.100397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:37.100452 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:37.100584 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:37.101781 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:37.101788 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:37.101794 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:37.101816 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:37.101870 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:37.101889 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:37.102032 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:37.102144 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 20:27:37.102219 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 20:27:37.110517 21587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1117d7342a3b443bb371ea4ad77b8165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37807 } }
I20260620 20:27:37.110919 21587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:37.111029 21587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1117d7342a3b443bb371ea4ad77b8165, State: Initialized, Role: FOLLOWER
I20260620 20:27:37.111698 21587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1117d7342a3b443bb371ea4ad77b8165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37807 } }
I20260620 20:27:37.111999 21587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:37.112170 21587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:37.112366 21587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:37.114636 21587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1117d7342a3b443bb371ea4ad77b8165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37807 } }
I20260620 20:27:37.115139 21587 leader_election.cc:302] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1117d7342a3b443bb371ea4ad77b8165; no voters:
I20260620 20:27:37.115767 21587 leader_election.cc:288] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:37.115978 21597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:37.116711 21597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165 [term 1 LEADER]: Becoming Leader. State: Replica: 1117d7342a3b443bb371ea4ad77b8165, State: Running, Role: LEADER
I20260620 20:27:37.117410 21597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1117d7342a3b443bb371ea4ad77b8165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37807 } }
I20260620 20:27:37.118098 21587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:37.120440 21598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1117d7342a3b443bb371ea4ad77b8165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1117d7342a3b443bb371ea4ad77b8165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37807 } } }
I20260620 20:27:37.120886 21598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:37.121475 21599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1117d7342a3b443bb371ea4ad77b8165. Latest consensus state: current_term: 1 leader_uuid: "1117d7342a3b443bb371ea4ad77b8165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1117d7342a3b443bb371ea4ad77b8165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37807 } } }
I20260620 20:27:37.121872 21599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:37.131017 21604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:37.136323 21604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:37.145955 21604 catalog_manager.cc:1347] Generated new cluster ID: 2dd64652c2414b0bb0105bd4201b51cd
I20260620 20:27:37.146114 21604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:37.192750 21604 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:37.193280 21604 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:37.202209 21604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1117d7342a3b443bb371ea4ad77b8165: Generated new TSK 0
I20260620 20:27:37.203250 21604 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:37.217137 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 21520
W20260620 20:27:37.218341 21582 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.242:33539
W20260620 20:27:37.219110 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 21520: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:27:37.219296 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 21520
2026-06-20T20:27:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1947 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:27:37 dist-test-slave-zzwf krb5kdc[21619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:27:37 dist-test-slave-zzwf krb5kdc[21619](info): set up 2 sockets
Jun 20 20:27:37 dist-test-slave-zzwf krb5kdc[21619](info): commencing operation
krb5kdc: starting...
W20260620 20:27:38.346418 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:27:38 dist-test-slave-zzwf krb5kdc[21619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:27:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:27:38.612552 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39323
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:38.950031 21632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:38.950314 21632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:38.950495 21632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:38.957850 21632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:38.957952 21632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:38.958086 21632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:38.958189 21632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:38.958261 21632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:38.968091 21632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39323
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:38.969393 21632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:38.970681 21632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:38.982858 21637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:38.982795 21640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:38.982762 21638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:38.983563 21632 server_base.cc:1034] running on GCE node
I20260620 20:27:38.984308 21632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:38.985757 21632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:38.986964 21632 hybrid_clock.cc:648] HybridClock initialized: now 1781987258986921 us; error 31 us; skew 500 ppm
I20260620 20:27:38.987322 21632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:38 dist-test-slave-zzwf krb5kdc[21619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:38.990540 21632 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:27:38.992090 21632 webserver.cc:460] Webserver started at http://127.0.0.1:41703/ using document root <none> and password file <none>
I20260620 20:27:38.992770 21632 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:38.992925 21632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:38.993233 21632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:38.996352 21632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "72fe7badfa5b4ce187bd6815379b2b91"
format_stamp: "Formatted at 2026-06-20 20:27:38 on dist-test-slave-zzwf"
server_key: "789b31f09052a1ff1d6325fc1588bd85"
server_key_iv: "52f7676592ca90a0080380c7aee0a419"
server_key_version: "encryptionkey@0"
I20260620 20:27:38.997130 21632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "72fe7badfa5b4ce187bd6815379b2b91"
format_stamp: "Formatted at 2026-06-20 20:27:38 on dist-test-slave-zzwf"
server_key: "789b31f09052a1ff1d6325fc1588bd85"
server_key_iv: "52f7676592ca90a0080380c7aee0a419"
server_key_version: "encryptionkey@0"
I20260620 20:27:39.002332 21632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 20:27:39.005789 21647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:39.007050 21632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:27:39.007237 21632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "72fe7badfa5b4ce187bd6815379b2b91"
format_stamp: "Formatted at 2026-06-20 20:27:38 on dist-test-slave-zzwf"
server_key: "789b31f09052a1ff1d6325fc1588bd85"
server_key_iv: "52f7676592ca90a0080380c7aee0a419"
server_key_version: "encryptionkey@0"
I20260620 20:27:39.007448 21632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:39.025800 21632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:39.030233 21632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:39.030529 21632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:39.050573 21632 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39323
I20260620 20:27:39.050630 21698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39323 every 8 connection(s)
I20260620 20:27:39.052124 21632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:39.055845 21699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:39.058661 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 21632
I20260620 20:27:39.058861 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:39.059407 32686 external_mini_cluster.cc:1428] Setting key 52b11bdaba788bd537490fd63fa297af
I20260620 20:27:39.069516 21699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91: Bootstrap starting.
I20260620 20:27:39.073120 21699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:39.074193 21699 log.cc:824] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91: Log is configured to *not* fsync() on all Append() calls
Jun 20 20:27:39 dist-test-slave-zzwf krb5kdc[21619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 20:27:39.077443 21699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91: No bootstrap required, opened a new log
I20260620 20:27:39.086050 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:39.061656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:59171 (local address 127.0.0.1:39323)
0620 20:27:39.062583 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:39.062598 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:39.062986 (+ 388us) server_negotiation.cc:407] Connection header received
0620 20:27:39.065245 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:39.065278 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:39.065859 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:39.066313 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:39.067640 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:39.069633 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:39.070925 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:39.072016 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:39.072204 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:39.076695 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:39.076706 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:39.076724 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:39.076770 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:39.080946 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:39.081923 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:39.081931 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:39.081940 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:39.082153 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:39.082643 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:39.082648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:39.082650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:39.083465 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:39.083786 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:39.084261 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 20:27:39.084626 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":391,"threads_started":1}
I20260620 20:27:39.088009 21699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72fe7badfa5b4ce187bd6815379b2b91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39323 } }
I20260620 20:27:39.088385 21699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:39.088500 21699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72fe7badfa5b4ce187bd6815379b2b91, State: Initialized, Role: FOLLOWER
I20260620 20:27:39.089061 21699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72fe7badfa5b4ce187bd6815379b2b91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39323 } }
I20260620 20:27:39.089349 21699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:39.089597 21699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:39.089798 21699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:39.090206 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:27:39.092046 21699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72fe7badfa5b4ce187bd6815379b2b91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39323 } }
I20260620 20:27:39.092535 21699 leader_election.cc:302] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72fe7badfa5b4ce187bd6815379b2b91; no voters:
I20260620 20:27:39.093266 21699 leader_election.cc:288] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:39.093469 21706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:39.094782 21706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91 [term 1 LEADER]: Becoming Leader. State: Replica: 72fe7badfa5b4ce187bd6815379b2b91, State: Running, Role: LEADER
I20260620 20:27:39.095464 21706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72fe7badfa5b4ce187bd6815379b2b91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39323 } }
I20260620 20:27:39.095747 21699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:39.099488 21709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72fe7badfa5b4ce187bd6815379b2b91. Latest consensus state: current_term: 1 leader_uuid: "72fe7badfa5b4ce187bd6815379b2b91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72fe7badfa5b4ce187bd6815379b2b91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39323 } } }
I20260620 20:27:39.099884 21709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:39.101444 21708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72fe7badfa5b4ce187bd6815379b2b91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72fe7badfa5b4ce187bd6815379b2b91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39323 } } }
I20260620 20:27:39.101848 21708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:39.106421 21715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:39.113150 21715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:39.116698 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:39.095601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:37203 (local address 127.0.0.1:39323)
0620 20:27:39.095793 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:39.095801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:39.096313 (+ 512us) server_negotiation.cc:407] Connection header received
0620 20:27:39.096491 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:39.096496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:39.096578 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:39.096706 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:39.098909 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:39.100199 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:39.102634 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:39.103594 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:39.103739 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:39.107442 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:39.107450 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:39.107464 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:39.107517 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:39.111927 (+ 4410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:39.113102 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:39.113108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:39.113112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:39.113358 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:39.114020 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:39.114025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:39.114028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:39.114489 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:39.115917 (+ 1428us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:39.116099 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 20:27:39.116305 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 20:27:39.125633 21715 catalog_manager.cc:1347] Generated new cluster ID: 76096e82a9fb4978b882f2f9f1c8b5f3
I20260620 20:27:39.125790 21715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:39.167856 21715 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:39.169801 21715 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:39.178256 21715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72fe7badfa5b4ce187bd6815379b2b91: Generated new TSK 0
I20260620 20:27:39.179241 21715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:39.184381 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 21632
I20260620 20:27:39.185830 21694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:39.281378 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 21632
W20260620 20:27:39.298890 21617 connection.cc:535] client connection to 127.0.0.1:39323 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39323): Connection reset by peer (error 104)
2026-06-20T20:27:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2062 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:27:39 dist-test-slave-zzwf krb5kdc[21731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:27:39 dist-test-slave-zzwf krb5kdc[21731](info): set up 2 sockets
Jun 20 20:27:39 dist-test-slave-zzwf krb5kdc[21731](info): commencing operation
krb5kdc: starting...
W20260620 20:27:40.414317 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:27:40 dist-test-slave-zzwf krb5kdc[21731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:27:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:27:40.684826 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:41.032809 21744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:41.033110 21744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:41.033322 21744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:41.041441 21744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:41.041630 21744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:41.041738 21744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:41.041831 21744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:41.041914 21744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:41.051963 21744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35539
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:41.053297 21744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:41.054679 21744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:41.066689 21749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:41.066758 21752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:41.066663 21750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:41.067651 21744 server_base.cc:1034] running on GCE node
I20260620 20:27:41.068569 21744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:41.070114 21744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:41.071370 21744 hybrid_clock.cc:648] HybridClock initialized: now 1781987261071304 us; error 54 us; skew 500 ppm
I20260620 20:27:41.071759 21744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:41 dist-test-slave-zzwf krb5kdc[21731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:41.075085 21744 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:27:41.076558 21744 webserver.cc:460] Webserver started at http://127.0.0.1:35225/ using document root <none> and password file <none>
I20260620 20:27:41.077226 21744 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:41.077337 21744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:41.077677 21744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:41.080755 21744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "a0dddb05a104444586885cfe1eaf2416"
format_stamp: "Formatted at 2026-06-20 20:27:41 on dist-test-slave-zzwf"
server_key: "10e661c98c0425229a4db59a4e53e533"
server_key_iv: "b6fefbc0dd55d08c54a1daeabd4e4876"
server_key_version: "encryptionkey@0"
I20260620 20:27:41.081561 21744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "a0dddb05a104444586885cfe1eaf2416"
format_stamp: "Formatted at 2026-06-20 20:27:41 on dist-test-slave-zzwf"
server_key: "10e661c98c0425229a4db59a4e53e533"
server_key_iv: "b6fefbc0dd55d08c54a1daeabd4e4876"
server_key_version: "encryptionkey@0"
I20260620 20:27:41.086829 21744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 20:27:41.090466 21759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:41.091910 21744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 20:27:41.092108 21744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "a0dddb05a104444586885cfe1eaf2416"
format_stamp: "Formatted at 2026-06-20 20:27:41 on dist-test-slave-zzwf"
server_key: "10e661c98c0425229a4db59a4e53e533"
server_key_iv: "b6fefbc0dd55d08c54a1daeabd4e4876"
server_key_version: "encryptionkey@0"
I20260620 20:27:41.092360 21744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:41.115464 21744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:41.120069 21744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:41.120383 21744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:41.139299 21744 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35539
I20260620 20:27:41.139346 21810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35539 every 8 connection(s)
I20260620 20:27:41.140759 21744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:41.144690 21811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:41.150192 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 21744
I20260620 20:27:41.150403 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:41.150835 32686 external_mini_cluster.cc:1428] Setting key 3acc4be3a62e0f08b0679fb06479cf19
I20260620 20:27:41.155701 21811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416: Bootstrap starting.
I20260620 20:27:41.159394 21811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:41.160564 21811 log.cc:824] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:41.163756 21811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416: No bootstrap required, opened a new log
I20260620 20:27:41.164526 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:41.152781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:53459 (local address 127.0.0.1:35539)
0620 20:27:41.153675 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:41.153689 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:41.153732 (+ 43us) server_negotiation.cc:407] Connection header received
0620 20:27:41.155421 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:41.155443 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:41.155885 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:41.156198 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:41.156838 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:41.158215 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:41.160051 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:41.160735 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:41.160876 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:41.162328 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:41.162338 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:41.162357 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:41.162870 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:41.162941 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:41.162965 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:41.163083 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:41.163374 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 20:27:41.163578 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":411,"threads_started":1}
I20260620 20:27:41.168615 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:27:41.172467 21811 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dddb05a104444586885cfe1eaf2416" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35539 } }
I20260620 20:27:41.172946 21811 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:41.173141 21811 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0dddb05a104444586885cfe1eaf2416, State: Initialized, Role: FOLLOWER
I20260620 20:27:41.173866 21811 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dddb05a104444586885cfe1eaf2416" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35539 } }
I20260620 20:27:41.174173 21811 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:41.174335 21811 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:41.174525 21811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:41.177778 21811 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dddb05a104444586885cfe1eaf2416" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35539 } }
I20260620 20:27:41.178418 21811 leader_election.cc:302] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0dddb05a104444586885cfe1eaf2416; no voters:
I20260620 20:27:41.179144 21811 leader_election.cc:288] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:41.179261 21821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:41.180320 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:41.172834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:53669 (local address 127.0.0.1:35539)
0620 20:27:41.173086 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:41.173094 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:41.173481 (+ 387us) server_negotiation.cc:407] Connection header received
0620 20:27:41.173692 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:41.173696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:41.173757 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:41.173852 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:41.174529 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:41.175334 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:41.177167 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:41.177704 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:41.177799 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:41.179106 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:41.179111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:41.179116 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:41.179145 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:41.179214 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:41.179239 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:41.179379 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:41.179947 (+ 568us) server_negotiation.cc:299] Negotiation successful
0620 20:27:41.180000 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 20:27:41.181232 21821 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416 [term 1 LEADER]: Becoming Leader. State: Replica: a0dddb05a104444586885cfe1eaf2416, State: Running, Role: LEADER
I20260620 20:27:41.181929 21811 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:41.181931 21821 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dddb05a104444586885cfe1eaf2416" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35539 } }
I20260620 20:27:41.185052 21822 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0dddb05a104444586885cfe1eaf2416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dddb05a104444586885cfe1eaf2416" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35539 } } }
I20260620 20:27:41.185526 21822 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:41.186151 21823 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0dddb05a104444586885cfe1eaf2416. Latest consensus state: current_term: 1 leader_uuid: "a0dddb05a104444586885cfe1eaf2416" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0dddb05a104444586885cfe1eaf2416" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35539 } } }
I20260620 20:27:41.186527 21823 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:41.186754 21830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:41.199527 21830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:41.211508 21830 catalog_manager.cc:1347] Generated new cluster ID: bc0483d596cd4fbca622a26bce835ebb
I20260620 20:27:41.211637 21830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:41.236996 21830 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:41.239127 21830 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:41.254029 21830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0dddb05a104444586885cfe1eaf2416: Generated new TSK 0
I20260620 20:27:41.254909 21830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:41.272658 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 21744
W20260620 20:27:41.273823 21806 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.242:53459
W20260620 20:27:41.274582 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 21744: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:27:41.274770 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 21744
W20260620 20:27:41.291891 21729 connection.cc:535] client connection to 127.0.0.1:35539 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35539): Connection reset by peer (error 104)
2026-06-20T20:27:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1992 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:27:41 dist-test-slave-zzwf krb5kdc[21843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:27:41 dist-test-slave-zzwf krb5kdc[21843](info): set up 2 sockets
Jun 20 20:27:41 dist-test-slave-zzwf krb5kdc[21843](info): commencing operation
krb5kdc: starting...
W20260620 20:27:42.410368 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:27:42 dist-test-slave-zzwf krb5kdc[21843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:27:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 20:27:42.689420 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43553
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:43.020999 21856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:43.021296 21856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:43.021464 21856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:43.028923 21856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:43.029034 21856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:43.029120 21856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:43.029202 21856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:43.029287 21856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:43.039526 21856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43553
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:43.040834 21856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:43.042162 21856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:43.053828 21861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:43.053951 21862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:43.054301 21864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:43.055195 21856 server_base.cc:1034] running on GCE node
I20260620 20:27:43.056037 21856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:43.057592 21856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:43.058840 21856 hybrid_clock.cc:648] HybridClock initialized: now 1781987263058787 us; error 47 us; skew 500 ppm
I20260620 20:27:43.059216 21856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:43 dist-test-slave-zzwf krb5kdc[21843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:43.062345 21856 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 20:27:43.063916 21856 webserver.cc:460] Webserver started at http://127.0.0.1:44157/ using document root <none> and password file <none>
I20260620 20:27:43.064602 21856 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:43.064752 21856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:43.065059 21856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:43.068138 21856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "3b622ab4cea7404fa587919f19fb015f"
format_stamp: "Formatted at 2026-06-20 20:27:43 on dist-test-slave-zzwf"
server_key: "df42ee94859d8c38452f82ee36c22dab"
server_key_iv: "571ba2569e066e3526b9e00756b1abd9"
server_key_version: "encryptionkey@0"
I20260620 20:27:43.068914 21856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "3b622ab4cea7404fa587919f19fb015f"
format_stamp: "Formatted at 2026-06-20 20:27:43 on dist-test-slave-zzwf"
server_key: "df42ee94859d8c38452f82ee36c22dab"
server_key_iv: "571ba2569e066e3526b9e00756b1abd9"
server_key_version: "encryptionkey@0"
I20260620 20:27:43.074251 21856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 20:27:43.077963 21871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:43.079490 21856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 20:27:43.079721 21856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "3b622ab4cea7404fa587919f19fb015f"
format_stamp: "Formatted at 2026-06-20 20:27:43 on dist-test-slave-zzwf"
server_key: "df42ee94859d8c38452f82ee36c22dab"
server_key_iv: "571ba2569e066e3526b9e00756b1abd9"
server_key_version: "encryptionkey@0"
I20260620 20:27:43.079960 21856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:43.086041 21856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:43.090520 21856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:43.090853 21856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:43.110828 21856 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43553
I20260620 20:27:43.110881 21922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43553 every 8 connection(s)
I20260620 20:27:43.112416 21856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:43.114356 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 21856
I20260620 20:27:43.114547 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:43.115139 32686 external_mini_cluster.cc:1428] Setting key f568c4beafb7a6126f05a8c41ce80781
I20260620 20:27:43.117218 21923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:43.123201 21926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:43.117013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:40823 (local address 127.0.0.1:43553)
0620 20:27:43.117912 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:43.117930 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:43.117974 (+ 44us) server_negotiation.cc:407] Connection header received
0620 20:27:43.119642 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:43.119664 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:43.120029 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:43.120365 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:43.121545 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:43.121557 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:43.121574 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:43.121590 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:43.121670 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:43.121694 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:43.121877 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:43.122002 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 20:27:43.122230 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":465,"threads_started":1}
I20260620 20:27:43.126942 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:27:43.130252 21923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f: Bootstrap starting.
I20260620 20:27:43.134613 21923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:43.134824 21926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:43.132284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:58653 (local address 127.0.0.1:43553)
0620 20:27:43.132451 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:43.132456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:43.132750 (+ 294us) server_negotiation.cc:407] Connection header received
0620 20:27:43.132971 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:43.132975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:43.133022 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:43.133096 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:43.134086 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:43.134091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:43.134094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:43.134109 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:43.134149 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:43.134164 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:43.134254 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:43.134391 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 20:27:43.134443 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 20:27:43.136080 21923 log.cc:824] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:43.139560 21923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f: No bootstrap required, opened a new log
I20260620 20:27:43.147399 21923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b622ab4cea7404fa587919f19fb015f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43553 } }
I20260620 20:27:43.147814 21923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:43.147948 21923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b622ab4cea7404fa587919f19fb015f, State: Initialized, Role: FOLLOWER
I20260620 20:27:43.148550 21923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b622ab4cea7404fa587919f19fb015f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43553 } }
I20260620 20:27:43.148800 21923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:43.148976 21923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:43.149168 21923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:43.151609 21923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b622ab4cea7404fa587919f19fb015f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43553 } }
I20260620 20:27:43.152173 21923 leader_election.cc:302] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b622ab4cea7404fa587919f19fb015f; no voters:
I20260620 20:27:43.152922 21923 leader_election.cc:288] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:43.153079 21933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:43.153975 21933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f [term 1 LEADER]: Becoming Leader. State: Replica: 3b622ab4cea7404fa587919f19fb015f, State: Running, Role: LEADER
I20260620 20:27:43.154600 21933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b622ab4cea7404fa587919f19fb015f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43553 } }
I20260620 20:27:43.155282 21923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:43.157662 21935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b622ab4cea7404fa587919f19fb015f. Latest consensus state: current_term: 1 leader_uuid: "3b622ab4cea7404fa587919f19fb015f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b622ab4cea7404fa587919f19fb015f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43553 } } }
I20260620 20:27:43.158082 21935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:43.158617 21934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b622ab4cea7404fa587919f19fb015f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b622ab4cea7404fa587919f19fb015f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43553 } } }
I20260620 20:27:43.158911 21934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:43.161257 21941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:43.166966 21941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:43.179491 21941 catalog_manager.cc:1347] Generated new cluster ID: c4e2b7f9b4bb49bfbcabc02f1bf249cf
I20260620 20:27:43.179625 21941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:43.196081 21941 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:43.196591 21941 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:43.205685 21941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b622ab4cea7404fa587919f19fb015f: Generated new TSK 0
I20260620 20:27:43.206524 21941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:43.217298 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 21856
W20260620 20:27:43.218511 21918 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.242:40823
W20260620 20:27:43.219326 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 21856: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:27:43.219482 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 21856
2026-06-20T20:27:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1944 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:27:43 dist-test-slave-zzwf krb5kdc[21954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:27:43 dist-test-slave-zzwf krb5kdc[21954](info): set up 2 sockets
Jun 20 20:27:43 dist-test-slave-zzwf krb5kdc[21954](info): commencing operation
krb5kdc: starting...
W20260620 20:27:44.334219 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:27:44 dist-test-slave-zzwf krb5kdc[21954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:27:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:27:44.601024 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:42591
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:42591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:44.941931 21967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:44.942227 21967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:44.942431 21967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:44.950048 21967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:44.950165 21967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:44.950286 21967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:44.950397 21967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:44.950479 21967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:44.960742 21967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:42591
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:42591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:44.962145 21967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:44.963483 21967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:44.975581 21972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:44.975498 21975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:44.975544 21973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:44.976468 21967 server_base.cc:1034] running on GCE node
I20260620 20:27:44.977227 21967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:44.978713 21967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:44.979964 21967 hybrid_clock.cc:648] HybridClock initialized: now 1781987264979914 us; error 46 us; skew 500 ppm
I20260620 20:27:44.980331 21967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:44 dist-test-slave-zzwf krb5kdc[21954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987264, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:44.983608 21967 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:27:44.985244 21967 webserver.cc:460] Webserver started at http://127.31.235.190:32989/ using document root <none> and password file <none>
I20260620 20:27:44.986063 21967 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:44.986215 21967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:44.986522 21967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:44.989579 21967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "89fc144a7aac46189a5085fb8721c110"
format_stamp: "Formatted at 2026-06-20 20:27:44 on dist-test-slave-zzwf"
server_key: "3e86cd658a7b266919160fe34076e031"
server_key_iv: "99ce020983c53980342519548a43e501"
server_key_version: "encryptionkey@0"
I20260620 20:27:44.990381 21967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "89fc144a7aac46189a5085fb8721c110"
format_stamp: "Formatted at 2026-06-20 20:27:44 on dist-test-slave-zzwf"
server_key: "3e86cd658a7b266919160fe34076e031"
server_key_iv: "99ce020983c53980342519548a43e501"
server_key_version: "encryptionkey@0"
I20260620 20:27:44.995682 21967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 20:27:44.999387 21982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:45.000897 21967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 20:27:45.001101 21967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "89fc144a7aac46189a5085fb8721c110"
format_stamp: "Formatted at 2026-06-20 20:27:44 on dist-test-slave-zzwf"
server_key: "3e86cd658a7b266919160fe34076e031"
server_key_iv: "99ce020983c53980342519548a43e501"
server_key_version: "encryptionkey@0"
I20260620 20:27:45.001310 21967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:45.017158 21967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:45.021881 21967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:45.022213 21967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:45.042021 21967 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:42591
I20260620 20:27:45.042039 22033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:42591 every 8 connection(s)
I20260620 20:27:45.043432 21967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:45.046981 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 21967
I20260620 20:27:45.047194 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:45.047252 22034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:45.047674 32686 external_mini_cluster.cc:1428] Setting key 14ace74fa0510c43333c25c96a5cca1b
I20260620 20:27:45.063022 22034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110: Bootstrap starting.
Jun 20 20:27:45 dist-test-slave-zzwf krb5kdc[21954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:27:45.067576 22034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:45.069144 22034 log.cc:824] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:45.073028 22034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110: No bootstrap required, opened a new log
I20260620 20:27:45.074966 22037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:45.049399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49797 (local address 127.31.235.190:42591)
0620 20:27:45.050319 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:45.050331 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:45.050373 (+ 42us) server_negotiation.cc:407] Connection header received
0620 20:27:45.052129 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:45.052151 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:45.052544 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:45.052867 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:45.054720 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:45.056316 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:45.058149 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:45.058938 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:45.059088 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:45.064592 (+ 5504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:45.064605 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:45.064628 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:45.064704 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:45.069247 (+ 4543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:45.070545 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:45.070553 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:45.070559 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:45.070832 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:45.071377 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:45.071382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:45.071386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:45.072162 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:45.072516 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:45.073058 (+ 542us) server_negotiation.cc:299] Negotiation successful
0620 20:27:45.073425 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":403,"threads_started":1}
I20260620 20:27:45.079571 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:27:45.083187 22034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc144a7aac46189a5085fb8721c110" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42591 } }
I20260620 20:27:45.083623 22034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:45.083781 22034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89fc144a7aac46189a5085fb8721c110, State: Initialized, Role: FOLLOWER
I20260620 20:27:45.084488 22034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc144a7aac46189a5085fb8721c110" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42591 } }
I20260620 20:27:45.084743 22034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:45.084930 22034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:45.085108 22034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:45.087992 22034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc144a7aac46189a5085fb8721c110" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42591 } }
I20260620 20:27:45.088718 22034 leader_election.cc:302] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89fc144a7aac46189a5085fb8721c110; no voters:
I20260620 20:27:45.089527 22034 leader_election.cc:288] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:45.089682 22044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:45.091895 22044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110 [term 1 LEADER]: Becoming Leader. State: Replica: 89fc144a7aac46189a5085fb8721c110, State: Running, Role: LEADER
I20260620 20:27:45.092026 22034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:45.092695 22044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc144a7aac46189a5085fb8721c110" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42591 } }
I20260620 20:27:45.096414 22045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89fc144a7aac46189a5085fb8721c110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc144a7aac46189a5085fb8721c110" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42591 } } }
I20260620 20:27:45.097010 22045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:45.098529 22046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89fc144a7aac46189a5085fb8721c110. Latest consensus state: current_term: 1 leader_uuid: "89fc144a7aac46189a5085fb8721c110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fc144a7aac46189a5085fb8721c110" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42591 } } }
I20260620 20:27:45.099782 22050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:45.099830 22046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:45.105758 22037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:45.084009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50529 (local address 127.31.235.190:42591)
0620 20:27:45.084248 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:45.084258 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:45.086115 (+ 1857us) server_negotiation.cc:407] Connection header received
0620 20:27:45.086261 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:45.086267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:45.086360 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:45.086489 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:45.087982 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:45.089206 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:45.091610 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:45.092411 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:45.092589 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:45.096082 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:45.096096 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:45.096101 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:45.096146 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:45.100544 (+ 4398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:45.101950 (+ 1406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:45.101963 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:45.101970 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:45.102169 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:45.102856 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:45.102862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:45.102866 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:45.103340 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:45.103670 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:45.105118 (+ 1448us) server_negotiation.cc:299] Negotiation successful
0620 20:27:45.105340 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:27:45.107519 22050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:45.117612 22050 catalog_manager.cc:1347] Generated new cluster ID: cbf9c06b97e9423d8082f9b9de8da8c0
I20260620 20:27:45.117859 22050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:45.129086 22050 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:45.130460 22050 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:45.140908 22050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89fc144a7aac46189a5085fb8721c110: Generated new TSK 0
I20260620 20:27:45.142179 22050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:45.159006 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 21967
I20260620 20:27:45.160769 22029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:45.256026 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 21967
W20260620 20:27:45.273370 21952 connection.cc:535] client connection to 127.31.235.190:42591 recv error: Network error: failed to read from TLS socket (remote: 127.31.235.190:42591): Connection reset by peer (error 104)
2026-06-20T20:27:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2037 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:27:45 dist-test-slave-zzwf krb5kdc[22066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:27:45 dist-test-slave-zzwf krb5kdc[22066](info): set up 2 sockets
Jun 20 20:27:45 dist-test-slave-zzwf krb5kdc[22066](info): commencing operation
krb5kdc: starting...
W20260620 20:27:46.434343 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:27:46 dist-test-slave-zzwf krb5kdc[22066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:27:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:27:46.695272 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:40557
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:37833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:40557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:47.032032 22079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:47.032330 22079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:47.032423 22079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:47.039670 22079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:47.039785 22079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:47.039832 22079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:47.039897 22079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:47.039943 22079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:47.049896 22079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:37833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:40557
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:40557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:47.051119 22079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:47.052388 22079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:47.064483 22084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:47.064416 22087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:47.064388 22085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:47.065356 22079 server_base.cc:1034] running on GCE node
I20260620 20:27:47.066143 22079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:47.067755 22079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:47.069005 22079 hybrid_clock.cc:648] HybridClock initialized: now 1781987267068954 us; error 42 us; skew 500 ppm
I20260620 20:27:47.069406 22079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:47 dist-test-slave-zzwf krb5kdc[22066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987267, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:47.072706 22079 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:27:47.074394 22079 webserver.cc:460] Webserver started at http://127.31.235.190:42861/ using document root <none> and password file <none>
I20260620 20:27:47.075114 22079 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:47.075268 22079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:47.075585 22079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:47.078701 22079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "c5c3b834ffda487e9e4c997b0e994915"
format_stamp: "Formatted at 2026-06-20 20:27:47 on dist-test-slave-zzwf"
server_key: "b6fe3a0392a6b9df9ef96ba76aa1334e"
server_key_iv: "4ec30cc1341b9e5110397e8bed7a4bef"
server_key_version: "encryptionkey@0"
I20260620 20:27:47.079463 22079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "c5c3b834ffda487e9e4c997b0e994915"
format_stamp: "Formatted at 2026-06-20 20:27:47 on dist-test-slave-zzwf"
server_key: "b6fe3a0392a6b9df9ef96ba76aa1334e"
server_key_iv: "4ec30cc1341b9e5110397e8bed7a4bef"
server_key_version: "encryptionkey@0"
I20260620 20:27:47.084712 22079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260620 20:27:47.088532 22094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:47.090068 22079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:27:47.090258 22079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "c5c3b834ffda487e9e4c997b0e994915"
format_stamp: "Formatted at 2026-06-20 20:27:47 on dist-test-slave-zzwf"
server_key: "b6fe3a0392a6b9df9ef96ba76aa1334e"
server_key_iv: "4ec30cc1341b9e5110397e8bed7a4bef"
server_key_version: "encryptionkey@0"
I20260620 20:27:47.090466 22079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:47.112699 22079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:47.117624 22079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:47.117939 22079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:47.137689 22079 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:40557
I20260620 20:27:47.137696 22145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:40557 every 8 connection(s)
I20260620 20:27:47.139134 22079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:47.141366 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 22079
I20260620 20:27:47.141590 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:47.142141 32686 external_mini_cluster.cc:1428] Setting key 9cd41029b88c93f5b4d3418d408b1964
I20260620 20:27:47.143982 22146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:47.156004 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:47.144131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57129 (local address 127.31.235.190:40557)
0620 20:27:47.145010 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:47.145020 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:47.145062 (+ 42us) server_negotiation.cc:407] Connection header received
0620 20:27:47.146715 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:47.146743 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:47.147113 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:47.147440 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:47.148042 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:47.149402 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:47.151357 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:47.152053 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:47.152188 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:47.153594 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:47.153604 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:47.153623 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:47.154240 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:47.154323 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:47.154355 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:47.154489 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:47.154783 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 20:27:47.155040 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":390,"threads_started":1}
I20260620 20:27:47.157230 22146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915: Bootstrap starting.
I20260620 20:27:47.159861 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:27:47.160630 22146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:47.161712 22146 log.cc:824] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:47.164858 22146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915: No bootstrap required, opened a new log
I20260620 20:27:47.171315 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:47.164332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33471 (local address 127.31.235.190:40557)
0620 20:27:47.164574 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:47.164580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:47.164957 (+ 377us) server_negotiation.cc:407] Connection header received
0620 20:27:47.165115 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:47.165119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:47.165175 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:47.165256 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:47.165908 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:47.166737 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:47.168627 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:47.169097 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:47.169198 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:47.170434 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:47.170439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:47.170442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:47.170458 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:47.170513 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:47.170531 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:47.170625 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:47.170893 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 20:27:47.170984 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:27:47.174582 22146 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c3b834ffda487e9e4c997b0e994915" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 40557 } }
I20260620 20:27:47.175020 22146 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:47.175143 22146 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5c3b834ffda487e9e4c997b0e994915, State: Initialized, Role: FOLLOWER
I20260620 20:27:47.175729 22146 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c3b834ffda487e9e4c997b0e994915" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 40557 } }
I20260620 20:27:47.175967 22146 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:47.176095 22146 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:47.176299 22146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:47.178654 22146 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c3b834ffda487e9e4c997b0e994915" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 40557 } }
I20260620 20:27:47.179224 22146 leader_election.cc:302] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5c3b834ffda487e9e4c997b0e994915; no voters:
I20260620 20:27:47.179790 22146 leader_election.cc:288] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:47.179932 22156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:47.180619 22156 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915 [term 1 LEADER]: Becoming Leader. State: Replica: c5c3b834ffda487e9e4c997b0e994915, State: Running, Role: LEADER
I20260620 20:27:47.181212 22156 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c3b834ffda487e9e4c997b0e994915" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 40557 } }
I20260620 20:27:47.182034 22146 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:47.184176 22158 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5c3b834ffda487e9e4c997b0e994915. Latest consensus state: current_term: 1 leader_uuid: "c5c3b834ffda487e9e4c997b0e994915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c3b834ffda487e9e4c997b0e994915" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 40557 } } }
I20260620 20:27:47.184684 22158 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:47.185528 22157 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5c3b834ffda487e9e4c997b0e994915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c3b834ffda487e9e4c997b0e994915" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 40557 } } }
I20260620 20:27:47.186049 22157 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:47.187238 22164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:47.192553 22164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:47.202800 22164 catalog_manager.cc:1347] Generated new cluster ID: 12aabbebc62848b98a3e2180ad33d430
I20260620 20:27:47.202992 22164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:47.222316 22164 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:47.224460 22164 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:47.243050 22164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5c3b834ffda487e9e4c997b0e994915: Generated new TSK 0
I20260620 20:27:47.244318 22164 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:47.255359 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 22079
W20260620 20:27:47.256635 22141 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57129
W20260620 20:27:47.257444 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 22079: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:27:47.257628 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 22079
W20260620 20:27:47.276320 22064 connection.cc:535] client connection to 127.31.235.190:40557 recv error: Network error: failed to read from TLS socket (remote: 127.31.235.190:40557): Connection reset by peer (error 104)
2026-06-20T20:27:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2002 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:27:47 dist-test-slave-zzwf krb5kdc[22177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:27:47 dist-test-slave-zzwf krb5kdc[22177](info): set up 2 sockets
Jun 20 20:27:47 dist-test-slave-zzwf krb5kdc[22177](info): commencing operation
krb5kdc: starting...
W20260620 20:27:48.370453 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:27:48 dist-test-slave-zzwf krb5kdc[22177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:27:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:27:48.637331 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:43575
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:43575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:48.976730 22190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:48.977053 22190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:48.977240 22190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:48.985266 22190 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:48.985383 22190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:48.985466 22190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:48.985597 22190 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:48.985673 22190 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:48.996285 22190 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:43575
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:43575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:48.997663 22190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:48.998971 22190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:49.011597 22195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:49.011549 22196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:49.011507 22198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:49.012606 22190 server_base.cc:1034] running on GCE node
I20260620 20:27:49.013338 22190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:49.014837 22190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:49.016048 22190 hybrid_clock.cc:648] HybridClock initialized: now 1781987269016000 us; error 64 us; skew 500 ppm
I20260620 20:27:49.016443 22190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:49 dist-test-slave-zzwf krb5kdc[22177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987269, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:49.027244 22190 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:27:49.028913 22190 webserver.cc:460] Webserver started at http://127.31.235.190:39013/ using document root <none> and password file <none>
I20260620 20:27:49.029701 22190 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:49.029829 22190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:49.030112 22190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:49.033371 22190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "23f56a30dab04246895acb68cf77dda2"
format_stamp: "Formatted at 2026-06-20 20:27:49 on dist-test-slave-zzwf"
server_key: "551a5f88714206ad8af7ea0262bf5b40"
server_key_iv: "0aa701063ea764cb942bf9e9151bc59b"
server_key_version: "encryptionkey@0"
I20260620 20:27:49.034224 22190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "23f56a30dab04246895acb68cf77dda2"
format_stamp: "Formatted at 2026-06-20 20:27:49 on dist-test-slave-zzwf"
server_key: "551a5f88714206ad8af7ea0262bf5b40"
server_key_iv: "0aa701063ea764cb942bf9e9151bc59b"
server_key_version: "encryptionkey@0"
I20260620 20:27:49.039897 22190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:27:49.043629 22205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:49.044996 22190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:27:49.045200 22190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "23f56a30dab04246895acb68cf77dda2"
format_stamp: "Formatted at 2026-06-20 20:27:49 on dist-test-slave-zzwf"
server_key: "551a5f88714206ad8af7ea0262bf5b40"
server_key_iv: "0aa701063ea764cb942bf9e9151bc59b"
server_key_version: "encryptionkey@0"
I20260620 20:27:49.045428 22190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:49.051625 22190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:49.055958 22190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:49.056284 22190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:49.077199 22256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:43575 every 8 connection(s)
I20260620 20:27:49.077168 22190 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:43575
I20260620 20:27:49.078851 22190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:49.082710 22257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:49.083359 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 22190
I20260620 20:27:49.083542 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:49.084131 32686 external_mini_cluster.cc:1428] Setting key 7f3075a25b682c87a0ddc0284895716a
I20260620 20:27:49.092978 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:49.086012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43563 (local address 127.31.235.190:43575)
0620 20:27:49.087074 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:49.087088 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:49.087137 (+ 49us) server_negotiation.cc:407] Connection header received
0620 20:27:49.089202 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:49.089233 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:49.089749 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:49.090145 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:49.091229 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:49.091240 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:49.091262 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:49.091282 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:49.091373 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:49.091414 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:49.091650 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:49.091822 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 20:27:49.092067 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":467,"threads_started":1}
I20260620 20:27:49.096873 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2: Bootstrap starting.
I20260620 20:27:49.097136 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:27:49.100103 22257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:49.101153 22257 log.cc:824] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:49.104602 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:49.101613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59819 (local address 127.31.235.190:43575)
0620 20:27:49.101843 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:49.101852 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:49.102470 (+ 618us) server_negotiation.cc:407] Connection header received
0620 20:27:49.102720 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:49.102726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:49.102790 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:49.102922 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:49.103881 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:49.103889 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:49.103894 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:49.103916 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:49.103965 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:49.103986 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:49.104120 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:49.104248 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 20:27:49.104315 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 20:27:49.104878 22257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2: No bootstrap required, opened a new log
I20260620 20:27:49.113103 22257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f56a30dab04246895acb68cf77dda2" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 43575 } }
I20260620 20:27:49.113588 22257 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:49.113698 22257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23f56a30dab04246895acb68cf77dda2, State: Initialized, Role: FOLLOWER
I20260620 20:27:49.114449 22257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f56a30dab04246895acb68cf77dda2" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 43575 } }
I20260620 20:27:49.114708 22257 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:49.114861 22257 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:49.115065 22257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:49.117268 22257 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f56a30dab04246895acb68cf77dda2" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 43575 } }
I20260620 20:27:49.117861 22257 leader_election.cc:302] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23f56a30dab04246895acb68cf77dda2; no voters:
I20260620 20:27:49.118544 22257 leader_election.cc:288] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:49.118700 22267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:49.119549 22267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2 [term 1 LEADER]: Becoming Leader. State: Replica: 23f56a30dab04246895acb68cf77dda2, State: Running, Role: LEADER
I20260620 20:27:49.120132 22267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f56a30dab04246895acb68cf77dda2" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 43575 } }
I20260620 20:27:49.120782 22257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:49.123268 22269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23f56a30dab04246895acb68cf77dda2. Latest consensus state: current_term: 1 leader_uuid: "23f56a30dab04246895acb68cf77dda2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f56a30dab04246895acb68cf77dda2" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 43575 } } }
I20260620 20:27:49.123730 22269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:49.124596 22276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:49.124496 22268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23f56a30dab04246895acb68cf77dda2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23f56a30dab04246895acb68cf77dda2" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 43575 } } }
I20260620 20:27:49.125559 22268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:49.130695 22276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:49.140136 22276 catalog_manager.cc:1347] Generated new cluster ID: ccf6dc93a2be48dbb080db331f9a351a
I20260620 20:27:49.140264 22276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:49.153717 22276 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:49.154296 22276 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:49.168891 22276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23f56a30dab04246895acb68cf77dda2: Generated new TSK 0
I20260620 20:27:49.169899 22276 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:49.182618 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 22190
W20260620 20:27:49.183806 22252 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43563
W20260620 20:27:49.184576 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 22190: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:27:49.184746 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 22190
2026-06-20T20:27:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1926 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:27:49 dist-test-slave-zzwf krb5kdc[22288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:27:49 dist-test-slave-zzwf krb5kdc[22288](info): set up 2 sockets
Jun 20 20:27:49 dist-test-slave-zzwf krb5kdc[22288](info): commencing operation
krb5kdc: starting...
W20260620 20:27:50.378952 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:27:50 dist-test-slave-zzwf krb5kdc[22288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:27:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:27:50.649807 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:34911
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:42491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:34911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:50.992561 22301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:50.992864 22301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:50.992960 22301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:51.000506 22301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:51.000639 22301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:51.000687 22301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:51.000746 22301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:51.000792 22301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:51.010978 22301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:42491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:34911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:34911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:51.012305 22301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:51.013602 22301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:51.025949 22306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:51.025990 22309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:51.026031 22307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:51.026788 22301 server_base.cc:1034] running on GCE node
I20260620 20:27:51.027659 22301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:51.029131 22301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:51.030402 22301 hybrid_clock.cc:648] HybridClock initialized: now 1781987271030311 us; error 46 us; skew 500 ppm
I20260620 20:27:51.030746 22301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:51 dist-test-slave-zzwf krb5kdc[22288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987271, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:51.034596 22301 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:27:51.036273 22301 webserver.cc:460] Webserver started at http://127.31.235.190:46351/ using document root <none> and password file <none>
I20260620 20:27:51.036978 22301 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:51.037082 22301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:51.037343 22301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:51.040465 22301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "b13ad4ca584c493aadae516e1a3f503c"
format_stamp: "Formatted at 2026-06-20 20:27:51 on dist-test-slave-zzwf"
server_key: "5cf71b33c6bd04a532f235091f677530"
server_key_iv: "44b32c8968531f7a334330d68e8e0ca8"
server_key_version: "encryptionkey@0"
I20260620 20:27:51.041216 22301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "b13ad4ca584c493aadae516e1a3f503c"
format_stamp: "Formatted at 2026-06-20 20:27:51 on dist-test-slave-zzwf"
server_key: "5cf71b33c6bd04a532f235091f677530"
server_key_iv: "44b32c8968531f7a334330d68e8e0ca8"
server_key_version: "encryptionkey@0"
I20260620 20:27:51.046679 22301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 20:27:51.050292 22316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:51.051756 22301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 20:27:51.051898 22301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "b13ad4ca584c493aadae516e1a3f503c"
format_stamp: "Formatted at 2026-06-20 20:27:51 on dist-test-slave-zzwf"
server_key: "5cf71b33c6bd04a532f235091f677530"
server_key_iv: "44b32c8968531f7a334330d68e8e0ca8"
server_key_version: "encryptionkey@0"
I20260620 20:27:51.052068 22301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:51.073195 22301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:51.078586 22301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:51.078909 22301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:51.100430 22301 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:34911
I20260620 20:27:51.100449 22367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:34911 every 8 connection(s)
I20260620 20:27:51.102051 22301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:51.105633 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 22301
I20260620 20:27:51.105849 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:51.106381 32686 external_mini_cluster.cc:1428] Setting key 76dd3119ec972e8f18d81f23354d5f1a
I20260620 20:27:51.106416 22368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:51.122016 22368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c: Bootstrap starting.
Jun 20 20:27:51 dist-test-slave-zzwf krb5kdc[22288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:27:51.125378 22368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:51.127024 22368 log.cc:824] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:51.130757 22368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c: No bootstrap required, opened a new log
I20260620 20:27:51.132997 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:51.108252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47311 (local address 127.31.235.190:34911)
0620 20:27:51.109152 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:51.109168 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:51.109218 (+ 50us) server_negotiation.cc:407] Connection header received
0620 20:27:51.111334 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:51.111360 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:51.111844 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:51.112280 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:51.114152 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:51.115618 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:51.117427 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:51.118328 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:51.118485 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:51.123817 (+ 5332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:51.123825 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:51.123848 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:51.123890 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:51.127936 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:51.128832 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:51.128842 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:51.128850 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:51.129071 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:51.129652 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:51.129657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:51.129660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:51.130387 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:51.130688 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:51.131118 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 20:27:51.131645 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":399,"threads_started":1}
I20260620 20:27:51.137470 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:27:51.140156 22368 raft_consensus.cc:348] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13ad4ca584c493aadae516e1a3f503c" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 34911 } }
I20260620 20:27:51.140722 22368 raft_consensus.cc:374] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:51.140857 22368 raft_consensus.cc:729] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b13ad4ca584c493aadae516e1a3f503c, State: Initialized, Role: FOLLOWER
I20260620 20:27:51.141557 22368 consensus_queue.cc:260] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13ad4ca584c493aadae516e1a3f503c" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 34911 } }
I20260620 20:27:51.141839 22368 raft_consensus.cc:388] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:51.141985 22368 raft_consensus.cc:482] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:51.142184 22368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:51.144807 22368 raft_consensus.cc:504] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13ad4ca584c493aadae516e1a3f503c" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 34911 } }
I20260620 20:27:51.145421 22368 leader_election.cc:302] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b13ad4ca584c493aadae516e1a3f503c; no voters:
I20260620 20:27:51.146205 22368 leader_election.cc:288] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:51.146307 22378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:51.148944 22378 raft_consensus.cc:686] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c [term 1 LEADER]: Becoming Leader. State: Replica: b13ad4ca584c493aadae516e1a3f503c, State: Running, Role: LEADER
I20260620 20:27:51.149739 22378 consensus_queue.cc:237] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13ad4ca584c493aadae516e1a3f503c" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 34911 } }
I20260620 20:27:51.150187 22368 sys_catalog.cc:564] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:51.153868 22379 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b13ad4ca584c493aadae516e1a3f503c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13ad4ca584c493aadae516e1a3f503c" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 34911 } } }
I20260620 20:27:51.153821 22380 sys_catalog.cc:455] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b13ad4ca584c493aadae516e1a3f503c. Latest consensus state: current_term: 1 leader_uuid: "b13ad4ca584c493aadae516e1a3f503c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b13ad4ca584c493aadae516e1a3f503c" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 34911 } } }
I20260620 20:27:51.154345 22379 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:51.154351 22380 sys_catalog.cc:458] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:51.155597 22383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:51.165606 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:51.142221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43269 (local address 127.31.235.190:34911)
0620 20:27:51.142443 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:51.142458 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:51.143123 (+ 665us) server_negotiation.cc:407] Connection header received
0620 20:27:51.143272 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:51.143279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:51.143375 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:51.143495 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:51.145062 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:51.145947 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:51.148181 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:51.149097 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:51.149246 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:51.152303 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:51.152310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:51.152315 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:51.152356 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:51.156479 (+ 4123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:51.162433 (+ 5954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:51.162442 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:51.162449 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:51.162637 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:51.163099 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:51.163106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:51.163109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:51.163571 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:51.163839 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:51.164498 (+ 659us) server_negotiation.cc:299] Negotiation successful
0620 20:27:51.165006 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:27:51.166450 22383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:51.179890 22383 catalog_manager.cc:1347] Generated new cluster ID: fef30142d67244278d9e06f47af614a3
I20260620 20:27:51.180039 22383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:51.192824 22383 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:51.194419 22383 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:51.204550 22383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b13ad4ca584c493aadae516e1a3f503c: Generated new TSK 0
I20260620 20:27:51.205586 22383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:51.211552 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 22301
I20260620 20:27:51.212954 22363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:51.304394 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 22301
2026-06-20T20:27:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2118 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:27:51 dist-test-slave-zzwf krb5kdc[22400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:27:51 dist-test-slave-zzwf krb5kdc[22400](info): set up 2 sockets
Jun 20 20:27:51 dist-test-slave-zzwf krb5kdc[22400](info): commencing operation
krb5kdc: starting...
W20260620 20:27:52.426426 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:27:52 dist-test-slave-zzwf krb5kdc[22400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:27:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:27:52.713737 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:37233
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:38667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:37233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:53.051352 22413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:53.051658 22413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:53.051847 22413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:53.059265 22413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:53.059394 22413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:53.059794 22413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:53.059921 22413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:53.060004 22413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:53.070374 22413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:38667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:37233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:37233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:53.071765 22413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:53.073117 22413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:53.085393 22418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:53.085438 22419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:53.085480 22421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:53.086513 22413 server_base.cc:1034] running on GCE node
I20260620 20:27:53.087314 22413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:53.088894 22413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:53.090142 22413 hybrid_clock.cc:648] HybridClock initialized: now 1781987273090108 us; error 58 us; skew 500 ppm
I20260620 20:27:53.090553 22413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:53 dist-test-slave-zzwf krb5kdc[22400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987273, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:53.093866 22413 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:27:53.095534 22413 webserver.cc:460] Webserver started at http://127.31.235.190:38223/ using document root <none> and password file <none>
I20260620 20:27:53.096205 22413 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:53.096319 22413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:53.096573 22413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:53.099768 22413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "8f2908d6b7be45e88fff39d7b0ce10b1"
format_stamp: "Formatted at 2026-06-20 20:27:53 on dist-test-slave-zzwf"
server_key: "dc4b1fcb72a9feac78a80545b46f6e93"
server_key_iv: "cd71219ca573f6b2a279a583d9725fff"
server_key_version: "encryptionkey@0"
I20260620 20:27:53.100642 22413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "8f2908d6b7be45e88fff39d7b0ce10b1"
format_stamp: "Formatted at 2026-06-20 20:27:53 on dist-test-slave-zzwf"
server_key: "dc4b1fcb72a9feac78a80545b46f6e93"
server_key_iv: "cd71219ca573f6b2a279a583d9725fff"
server_key_version: "encryptionkey@0"
I20260620 20:27:53.107842 22413 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260620 20:27:53.111759 22428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:53.113250 22413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:27:53.113420 22413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "8f2908d6b7be45e88fff39d7b0ce10b1"
format_stamp: "Formatted at 2026-06-20 20:27:53 on dist-test-slave-zzwf"
server_key: "dc4b1fcb72a9feac78a80545b46f6e93"
server_key_iv: "cd71219ca573f6b2a279a583d9725fff"
server_key_version: "encryptionkey@0"
I20260620 20:27:53.113631 22413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:53.137027 22413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:53.141304 22413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:53.141593 22413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:53.160946 22413 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:37233
I20260620 20:27:53.160980 22479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:37233 every 8 connection(s)
I20260620 20:27:53.162358 22413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:53.166497 22480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:53.170374 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 22413
I20260620 20:27:53.170579 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:53.171159 32686 external_mini_cluster.cc:1428] Setting key f66135e15883d48652822f6f9e4544b9
I20260620 20:27:53.179663 22480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1: Bootstrap starting.
I20260620 20:27:53.183257 22480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:53.184556 22480 log.cc:824] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:53.184693 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:53.173295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43501 (local address 127.31.235.190:37233)
0620 20:27:53.174144 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:53.174157 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:53.174198 (+ 41us) server_negotiation.cc:407] Connection header received
0620 20:27:53.175928 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:53.175951 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:53.176377 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:53.176797 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:53.177382 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:53.178769 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:53.180182 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:53.180838 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:53.181043 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:53.182362 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:53.182372 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:53.182390 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:53.182932 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:53.183006 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:53.183031 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:53.183173 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:53.183397 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 20:27:53.183672 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":397,"threads_started":1}
I20260620 20:27:53.187958 22480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1: No bootstrap required, opened a new log
I20260620 20:27:53.188911 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:27:53.197758 22480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2908d6b7be45e88fff39d7b0ce10b1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 37233 } }
I20260620 20:27:53.198331 22480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:53.198462 22480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2908d6b7be45e88fff39d7b0ce10b1, State: Initialized, Role: FOLLOWER
I20260620 20:27:53.199170 22480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2908d6b7be45e88fff39d7b0ce10b1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 37233 } }
I20260620 20:27:53.199491 22480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:53.199699 22480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:53.199936 22480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:53.200929 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:53.193428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54133 (local address 127.31.235.190:37233)
0620 20:27:53.193664 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:53.193676 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:53.194043 (+ 367us) server_negotiation.cc:407] Connection header received
0620 20:27:53.194206 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:53.194211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:53.194306 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:53.194430 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:53.194925 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:53.196239 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:53.197698 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:53.198412 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:53.198559 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:53.199872 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:53.199884 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:53.199890 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:53.199917 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:53.199977 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:53.200003 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:53.200117 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:53.200444 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 20:27:53.200547 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 20:27:53.203244 22480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2908d6b7be45e88fff39d7b0ce10b1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 37233 } }
I20260620 20:27:53.203820 22480 leader_election.cc:302] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f2908d6b7be45e88fff39d7b0ce10b1; no voters:
I20260620 20:27:53.204464 22480 leader_election.cc:288] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:53.204640 22490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:53.205909 22490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1 [term 1 LEADER]: Becoming Leader. State: Replica: 8f2908d6b7be45e88fff39d7b0ce10b1, State: Running, Role: LEADER
I20260620 20:27:53.206517 22490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2908d6b7be45e88fff39d7b0ce10b1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 37233 } }
I20260620 20:27:53.206954 22480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:53.209550 22491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f2908d6b7be45e88fff39d7b0ce10b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2908d6b7be45e88fff39d7b0ce10b1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 37233 } } }
I20260620 20:27:53.209959 22491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:53.209698 22492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f2908d6b7be45e88fff39d7b0ce10b1. Latest consensus state: current_term: 1 leader_uuid: "8f2908d6b7be45e88fff39d7b0ce10b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f2908d6b7be45e88fff39d7b0ce10b1" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 37233 } } }
I20260620 20:27:53.210713 22492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:53.218547 22498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:53.223690 22498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:53.232702 22498 catalog_manager.cc:1347] Generated new cluster ID: 65562fca194d4fba99618a2f4e7283dd
I20260620 20:27:53.232864 22498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:53.252988 22498 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:53.254427 22498 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:53.264842 22498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f2908d6b7be45e88fff39d7b0ce10b1: Generated new TSK 0
I20260620 20:27:53.265996 22498 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:53.274397 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 22413
W20260620 20:27:53.275521 22475 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43501
W20260620 20:27:53.276242 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 22413: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:27:53.276387 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 22413
2026-06-20T20:27:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1971 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:27:53 dist-test-slave-zzwf krb5kdc[22511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:27:53 dist-test-slave-zzwf krb5kdc[22511](info): set up 2 sockets
Jun 20 20:27:53 dist-test-slave-zzwf krb5kdc[22511](info): commencing operation
krb5kdc: starting...
W20260620 20:27:54.386350 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:27:54 dist-test-slave-zzwf krb5kdc[22511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:27:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:27:54.660418 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:46289
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:41911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:46289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:55.000703 22524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:55.001078 22524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:55.001255 22524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:55.009534 22524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:55.009701 22524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:55.009790 22524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:55.009881 22524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:55.009950 22524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:55.020244 22524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:41911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:46289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:46289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:55.021729 22524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:55.023057 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:55.035315 22530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:55.035415 22532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:55.035386 22529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:55.036506 22524 server_base.cc:1034] running on GCE node
I20260620 20:27:55.037372 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:55.038938 22524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:55.040190 22524 hybrid_clock.cc:648] HybridClock initialized: now 1781987275040165 us; error 57 us; skew 500 ppm
I20260620 20:27:55.040585 22524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:55 dist-test-slave-zzwf krb5kdc[22511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987275, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:55.049077 22524 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:27:55.050817 22524 webserver.cc:460] Webserver started at http://127.31.235.190:43535/ using document root <none> and password file <none>
I20260620 20:27:55.051561 22524 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:55.051719 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:55.052059 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:55.055198 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "bd7b6d22bd284ab2aa15bea50ce3ed9b"
format_stamp: "Formatted at 2026-06-20 20:27:55 on dist-test-slave-zzwf"
server_key: "73cbc7c320b3d45e6ed42ba561cca34c"
server_key_iv: "f05c626bde6e69c2d786af6492ac248a"
server_key_version: "encryptionkey@0"
I20260620 20:27:55.055968 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "bd7b6d22bd284ab2aa15bea50ce3ed9b"
format_stamp: "Formatted at 2026-06-20 20:27:55 on dist-test-slave-zzwf"
server_key: "73cbc7c320b3d45e6ed42ba561cca34c"
server_key_iv: "f05c626bde6e69c2d786af6492ac248a"
server_key_version: "encryptionkey@0"
I20260620 20:27:55.062367 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 20:27:55.066002 22539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:55.067409 22524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 20:27:55.067610 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "bd7b6d22bd284ab2aa15bea50ce3ed9b"
format_stamp: "Formatted at 2026-06-20 20:27:55 on dist-test-slave-zzwf"
server_key: "73cbc7c320b3d45e6ed42ba561cca34c"
server_key_iv: "f05c626bde6e69c2d786af6492ac248a"
server_key_version: "encryptionkey@0"
I20260620 20:27:55.067819 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:55.073854 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:55.077917 22524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:55.078217 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:55.097666 22524 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:46289
I20260620 20:27:55.097767 22590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:46289 every 8 connection(s)
I20260620 20:27:55.099113 22524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:55.103142 22591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:55.109615 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 22524
I20260620 20:27:55.109828 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:55.110371 32686 external_mini_cluster.cc:1428] Setting key 59e1ede90a99fe7444fe018f4be68966
I20260620 20:27:55.115137 22591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b: Bootstrap starting.
I20260620 20:27:55.118645 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:55.112334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45289 (local address 127.31.235.190:46289)
0620 20:27:55.113308 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:55.113320 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:55.113362 (+ 42us) server_negotiation.cc:407] Connection header received
0620 20:27:55.115104 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:55.115130 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:55.115525 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:55.115862 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:55.116979 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:55.116986 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:55.116998 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:55.117013 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:55.117078 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:55.117098 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:55.117309 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:55.117408 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 20:27:55.117695 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":496,"threads_started":1}
I20260620 20:27:55.118965 22591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:55.120078 22591 log.cc:824] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:55.122768 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:27:55.123765 22591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b: No bootstrap required, opened a new log
I20260620 20:27:55.129920 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:55.127287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42823 (local address 127.31.235.190:46289)
0620 20:27:55.127509 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:55.127514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:55.127850 (+ 336us) server_negotiation.cc:407] Connection header received
0620 20:27:55.128041 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:55.128044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:55.128096 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:55.128184 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:55.129142 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:55.129146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:55.129150 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 20:27:55.129163 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:55.129206 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 20:27:55.129221 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:55.129351 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:55.129474 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 20:27:55.129596 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:27:55.132946 22591 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b6d22bd284ab2aa15bea50ce3ed9b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46289 } }
I20260620 20:27:55.133368 22591 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 20:27:55.133481 22591 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd7b6d22bd284ab2aa15bea50ce3ed9b, State: Initialized, Role: FOLLOWER
I20260620 20:27:55.134104 22591 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b6d22bd284ab2aa15bea50ce3ed9b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46289 } }
I20260620 20:27:55.134385 22591 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 20:27:55.134547 22591 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 20:27:55.134750 22591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:55.137017 22591 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b6d22bd284ab2aa15bea50ce3ed9b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46289 } }
I20260620 20:27:55.137611 22591 leader_election.cc:302] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd7b6d22bd284ab2aa15bea50ce3ed9b; no voters:
I20260620 20:27:55.138505 22591 leader_election.cc:288] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 20:27:55.138609 22601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:55.139329 22601 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b [term 1 LEADER]: Becoming Leader. State: Replica: bd7b6d22bd284ab2aa15bea50ce3ed9b, State: Running, Role: LEADER
I20260620 20:27:55.139966 22601 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b6d22bd284ab2aa15bea50ce3ed9b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46289 } }
I20260620 20:27:55.141090 22591 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:55.143221 22603 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd7b6d22bd284ab2aa15bea50ce3ed9b. Latest consensus state: current_term: 1 leader_uuid: "bd7b6d22bd284ab2aa15bea50ce3ed9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b6d22bd284ab2aa15bea50ce3ed9b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46289 } } }
I20260620 20:27:55.143702 22603 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:55.144052 22602 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd7b6d22bd284ab2aa15bea50ce3ed9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd7b6d22bd284ab2aa15bea50ce3ed9b" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46289 } } }
I20260620 20:27:55.144697 22602 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:55.145383 22606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:55.150681 22606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:27:55.162930 22606 catalog_manager.cc:1347] Generated new cluster ID: 7d422b73b06a43529520804bd56ab7ed
I20260620 20:27:55.163091 22606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:55.225713 22606 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:55.226226 22606 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:55.238106 22606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd7b6d22bd284ab2aa15bea50ce3ed9b: Generated new TSK 0
I20260620 20:27:55.239003 22606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:55.253679 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 22524
W20260620 20:27:55.255005 22586 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45289
W20260620 20:27:55.255753 32686 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 22524: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 20:27:55.255910 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 22524
2026-06-20T20:27:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1979 ms)
[----------] 18 tests from AuthTokenIssuingTest (35789 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:27:55 dist-test-slave-zzwf krb5kdc[22622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:27:55 dist-test-slave-zzwf krb5kdc[22622](info): set up 2 sockets
Jun 20 20:27:55 dist-test-slave-zzwf krb5kdc[22622](info): commencing operation
krb5kdc: starting...
W20260620 20:27:56.378484 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:27:56 dist-test-slave-zzwf krb5kdc[22622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:27:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:27:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:27:56.646034 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:42077
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:42077,127.31.235.189:45071,127.31.235.188:37895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:27:56.981886 22635 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:27:56.982265 22635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:56.982407 22635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:56.982574 22635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:56.990021 22635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:56.990185 22635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:56.990295 22635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:56.990393 22635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:56.990509 22635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:57.000910 22635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:42077,127.31.235.189:45071,127.31.235.188:37895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:42077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:57.002308 22635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:57.003684 22635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:57.015741 22640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:57.015694 22643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:57.015722 22641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:57.016772 22635 server_base.cc:1034] running on GCE node
I20260620 20:27:57.017623 22635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:57.019107 22635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:57.020382 22635 hybrid_clock.cc:648] HybridClock initialized: now 1781987277020311 us; error 45 us; skew 500 ppm
I20260620 20:27:57.020809 22635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:57 dist-test-slave-zzwf krb5kdc[22622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987277, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:57.024048 22635 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:27:57.025559 22635 webserver.cc:460] Webserver started at http://127.31.235.190:44389/ using document root <none> and password file <none>
I20260620 20:27:57.026266 22635 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:57.026410 22635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:57.026731 22635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:57.029783 22635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "47aac6a2946647c192e482ea0350d716"
format_stamp: "Formatted at 2026-06-20 20:27:57 on dist-test-slave-zzwf"
server_key: "3f58148d83890bab7171550df45e52a0"
server_key_iv: "afb1f37b8edcf7419e104a598cdfc5a6"
server_key_version: "encryptionkey@0"
I20260620 20:27:57.030588 22635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "47aac6a2946647c192e482ea0350d716"
format_stamp: "Formatted at 2026-06-20 20:27:57 on dist-test-slave-zzwf"
server_key: "3f58148d83890bab7171550df45e52a0"
server_key_iv: "afb1f37b8edcf7419e104a598cdfc5a6"
server_key_version: "encryptionkey@0"
I20260620 20:27:57.035872 22635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:27:57.039599 22650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:57.041005 22635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:27:57.041201 22635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "47aac6a2946647c192e482ea0350d716"
format_stamp: "Formatted at 2026-06-20 20:27:57 on dist-test-slave-zzwf"
server_key: "3f58148d83890bab7171550df45e52a0"
server_key_iv: "afb1f37b8edcf7419e104a598cdfc5a6"
server_key_version: "encryptionkey@0"
I20260620 20:27:57.041453 22635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:57.057096 22635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:57.061781 22635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:57.062081 22635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:57.081473 22635 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:42077
I20260620 20:27:57.081514 22701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:42077 every 8 connection(s)
I20260620 20:27:57.083004 22635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:27:57.086755 22702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:57.092378 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 22635
I20260620 20:27:57.092700 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:27:57.093392 32686 external_mini_cluster.cc:1428] Setting key 15723ea7a9a321815b5b7f27de74788a
I20260620 20:27:57.094030 22702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42077 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:27:57 dist-test-slave-zzwf krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987277, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:27:57.121888 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.097471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:42077 (local address 127.0.0.1:40444)
0620 20:27:57.098685 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.098724 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:57.101274 (+ 2550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:57.102440 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:57.102449 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:57.103265 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:57.103619 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:57.103638 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:57.106301 (+ 2663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:57.106305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:57.107437 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:57.107450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:57.108871 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:57.108876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:57.109016 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:57.109777 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:27:57.109799 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:57.112119 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:57.117335 (+ 5216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:57.117346 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:57.117358 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:57.117879 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:57.118434 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:57.118438 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:57.118440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:57.118617 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:57.119829 (+ 1212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:57.119836 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:57.120058 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 20:27:57.120284 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 20:27:57.120717 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"mutex_wait_us":1,"thread_start_us":766,"threads_started":1}
I20260620 20:27:57.121941 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.097458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40444 (local address 127.31.235.190:42077)
0620 20:27:57.098357 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:57.098373 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:57.098855 (+ 482us) server_negotiation.cc:407] Connection header received
0620 20:27:57.101763 (+ 2908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:57.101775 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:57.102125 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:57.102386 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:57.103859 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:57.106118 (+ 2259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:57.107594 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:57.108639 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:57.108869 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:57.112398 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:57.112413 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:57.112436 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:57.112484 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:57.117111 (+ 4627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:57.118034 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:57.118046 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:57.118052 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:57.118287 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:57.118762 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:57.118770 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:57.118777 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:57.119658 (+ 881us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:57.119913 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:57.120496 (+ 583us) server_negotiation.cc:299] Negotiation successful
0620 20:27:57.120933 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":597,"spinlock_wait_cycles":20224,"thread_start_us":365,"threads_started":1}
WARNING: no policy specified for kudu/127.31.235.189@KRBTEST.COM; defaulting to no policy
I20260620 20:27:57.125022 22702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.189@KRBTEST.COM" created.
I20260620 20:27:57.126346 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.125898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:45071 (local address 127.0.0.1:41626)
0620 20:27:57.126076 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.126218 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 20:27:57.126638 22653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.235.189:45071: connect: Connection refused (error 111)
I20260620 20:27:57.128258 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.127891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:45071 (local address 127.0.0.1:41632)
0620 20:27:57.128036 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.128112 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 20:27:57.128479 22702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.189:45071: Network error: Client connection negotiation failed: client connection to 127.31.235.189:45071: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.189.keytab.
Entry for principal kudu/127.31.235.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.189.keytab.
I20260620 20:27:57.158090 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.189:45071
--webserver_interface=127.31.235.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.189.keytab
--principal=kudu/127.31.235.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:42077,127.31.235.189:45071,127.31.235.188:37895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:27:57.177795 22702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } attempt: 1
I20260620 20:27:57.179123 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.178494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:45071 (local address 127.0.0.1:41638)
0620 20:27:57.178759 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.178926 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 20:27:57.180326 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.179851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:45071 (local address 127.0.0.1:41650)
0620 20:27:57.180036 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.180134 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 20:27:57.180577 22702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.189:45071: Network error: Client connection negotiation failed: client connection to 127.31.235.189:45071: connect: Connection refused (error 111)
I20260620 20:27:57.249078 22702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } attempt: 2
I20260620 20:27:57.250728 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.249829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:45071 (local address 127.0.0.1:41656)
0620 20:27:57.250392 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.250523 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 20:27:57.252233 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.251546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:45071 (local address 127.0.0.1:41670)
0620 20:27:57.251966 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.252063 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271}
W20260620 20:27:57.252552 22702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.189:45071: Network error: Client connection negotiation failed: client connection to 127.31.235.189:45071: connect: Connection refused (error 111)
I20260620 20:27:57.343917 22702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } attempt: 3
I20260620 20:27:57.345161 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.344640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:45071 (local address 127.0.0.1:41674)
0620 20:27:57.344876 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.344985 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 20:27:57.346364 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.345940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:45071 (local address 127.0.0.1:41684)
0620 20:27:57.346101 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.346185 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 20:27:57.346549 22702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.189:45071: Network error: Client connection negotiation failed: client connection to 127.31.235.189:45071: connect: Connection refused (error 111)
I20260620 20:27:57.489782 22702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } attempt: 4
I20260620 20:27:57.491147 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.490510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:45071 (local address 127.0.0.1:41698)
0620 20:27:57.490842 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.490965 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260620 20:27:57.492311 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.491892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:45071 (local address 127.0.0.1:41702)
0620 20:27:57.492062 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.492157 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:45071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 20:27:57.492566 22702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.189:45071: Network error: Client connection negotiation failed: client connection to 127.31.235.189:45071: connect: Connection refused (error 111)
W20260620 20:27:57.501367 22709 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:27:57.501828 22709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:57.501920 22709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:57.502067 22709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:57.509581 22709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:57.509668 22709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:57.509761 22709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:57.509851 22709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:57.509939 22709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:57.520164 22709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:42077,127.31.235.189:45071,127.31.235.188:37895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.189.keytab
--principal=kudu/127.31.235.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.189:45071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.235.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:57.521478 22709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:57.522877 22709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:57.535262 22715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:57.535398 22717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:57.535379 22714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:57.536128 22709 server_base.cc:1034] running on GCE node
I20260620 20:27:57.536926 22709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:57.538487 22709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:57.539712 22709 hybrid_clock.cc:648] HybridClock initialized: now 1781987277539634 us; error 54 us; skew 500 ppm
I20260620 20:27:57.540071 22709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:57 dist-test-slave-zzwf krb5kdc[22622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987277, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:57.543314 22709 init.cc:377] Logged in from keytab as kudu/127.31.235.189@KRBTEST.COM (short username kudu)
I20260620 20:27:57.544788 22709 webserver.cc:460] Webserver started at http://127.31.235.189:45619/ using document root <none> and password file <none>
I20260620 20:27:57.545606 22709 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:57.545697 22709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:57.546021 22709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:57.548974 22709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-1/data/instance:
uuid: "33d34284525b406bb2a4da33f7763c29"
format_stamp: "Formatted at 2026-06-20 20:27:57 on dist-test-slave-zzwf"
server_key: "418ea0f9a42abcf740b79b707d3621de"
server_key_iv: "0f0a65faee2e63a9b3145d95d3865cb2"
server_key_version: "encryptionkey@0"
I20260620 20:27:57.549819 22709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-1/wal/instance:
uuid: "33d34284525b406bb2a4da33f7763c29"
format_stamp: "Formatted at 2026-06-20 20:27:57 on dist-test-slave-zzwf"
server_key: "418ea0f9a42abcf740b79b707d3621de"
server_key_iv: "0f0a65faee2e63a9b3145d95d3865cb2"
server_key_version: "encryptionkey@0"
I20260620 20:27:57.554749 22709 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 20:27:57.558274 22724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:57.559643 22709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:27:57.559861 22709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-1/wal
uuid: "33d34284525b406bb2a4da33f7763c29"
format_stamp: "Formatted at 2026-06-20 20:27:57 on dist-test-slave-zzwf"
server_key: "418ea0f9a42abcf740b79b707d3621de"
server_key_iv: "0f0a65faee2e63a9b3145d95d3865cb2"
server_key_version: "encryptionkey@0"
I20260620 20:27:57.560106 22709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:57.584133 22709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:57.588583 22709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:57.588899 22709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:57.609831 22709 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.189:45071
I20260620 20:27:57.609858 22775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.189:45071 every 8 connection(s)
I20260620 20:27:57.611346 22709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-1/data/info.pb
I20260620 20:27:57.613943 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 22709
I20260620 20:27:57.614126 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-1/wal/instance
I20260620 20:27:57.614660 32686 external_mini_cluster.cc:1428] Setting key 6ba48ad38e0096dd6a9db15a571c0bf4
I20260620 20:27:57.615942 22776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:57.626101 22776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42077 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:27:57 dist-test-slave-zzwf krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987277, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.189@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.235.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.188@KRBTEST.COM" created.
I20260620 20:27:57.648344 22779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.628934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40458 (local address 127.31.235.190:42077)
0620 20:27:57.629481 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:57.629534 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:57.630641 (+ 1107us) server_negotiation.cc:407] Connection header received
0620 20:27:57.632508 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:57.632513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:57.632607 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:57.632726 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:57.634334 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:57.635061 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:57.636588 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:57.636998 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:57.637133 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:57.641128 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:57.641133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:57.641136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:57.641181 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:57.644171 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:57.645074 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:57.645079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:57.645081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:57.645201 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:57.645839 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:57.645843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:57.645845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:57.646350 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:57.646570 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:57.647432 (+ 862us) server_negotiation.cc:299] Negotiation successful
0620 20:27:57.647702 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":281,"threads_started":1}
I20260620 20:27:57.648962 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.629679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:42077 (local address 127.0.0.1:40458)
0620 20:27:57.630478 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.630514 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:57.632234 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:57.632970 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:57.632979 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:57.633556 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:57.634119 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:57.634135 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:57.635232 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:57.635236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:57.636430 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:57.636443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:57.637242 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:57.637250 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:57.637485 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:57.638298 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:27:57.638332 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:57.640896 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:57.644327 (+ 3431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:57.644336 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:57.644355 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:57.644935 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:57.645397 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:57.645402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:57.645406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:57.645697 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:57.646550 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:57.646557 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:57.646962 (+ 405us) client_negotiation.cc:769] Sending connection context
0620 20:27:57.647242 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 20:27:57.647698 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":333,"threads_started":1}
I20260620 20:27:57.650905 22776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:27:57 dist-test-slave-zzwf krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987277, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.189@KRBTEST.COM for kudu/127.31.235.189@KRBTEST.COM
I20260620 20:27:57.668187 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.651803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:45071 (local address 127.0.0.1:41708)
0620 20:27:57.652021 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.652041 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:57.652168 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:57.653330 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:57.653334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:57.653800 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:57.653946 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:57.653955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:57.655750 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:57.655756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:57.657583 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:57.657603 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:57.658508 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:57.658517 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:57.658776 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:57.659429 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:27:57.659460 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:57.662192 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:57.665727 (+ 3535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:57.665734 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:57.665738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:57.666103 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:57.666478 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:57.666483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:57.666487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:57.666619 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:57.667203 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:57.667207 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:57.667358 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 20:27:57.667503 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 20:27:57.667792 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 20:27:57.668294 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.652021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41708 (local address 127.31.235.189:45071)
0620 20:27:57.652786 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:57.652796 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:57.652828 (+ 32us) server_negotiation.cc:407] Connection header received
0620 20:27:57.652883 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:57.652893 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:57.653130 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:57.653278 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:57.654151 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:57.655593 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:57.657774 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:57.658338 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:57.658467 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:57.662410 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:57.662421 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:57.662436 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:57.662467 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:57.665603 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:57.666242 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:57.666254 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:57.666256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:57.666364 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:57.666729 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:57.666734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:57.666737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:57.667083 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:57.667301 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:57.667650 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 20:27:57.667918 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":575,"thread_start_us":416,"threads_started":1}
I20260620 20:27:57.670759 22776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:27:57.672181 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.671537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:37895 (local address 127.0.0.1:38536)
0620 20:27:57.671783 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.671939 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260620 20:27:57.672480 22727 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
I20260620 20:27:57.674433 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.673992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:37895 (local address 127.0.0.1:38542)
0620 20:27:57.674161 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.674254 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 20:27:57.674705 22776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:37895: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.188.keytab.
Entry for principal kudu/127.31.235.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.188.keytab.
I20260620 20:27:57.684980 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.188:37895
--webserver_interface=127.31.235.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.188.keytab
--principal=kudu/127.31.235.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:42077,127.31.235.189:45071,127.31.235.188:37895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:27:57.724035 22776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } attempt: 1
I20260620 20:27:57.725255 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.724799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:37895 (local address 127.0.0.1:38554)
0620 20:27:57.724998 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.725106 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 20:27:57.726369 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.726019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:37895 (local address 127.0.0.1:38568)
0620 20:27:57.726168 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.726240 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 20:27:57.726636 22776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:37895: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
I20260620 20:27:57.791887 22702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } attempt: 5
I20260620 20:27:57.794869 22776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } attempt: 2
I20260620 20:27:57.795883 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.795417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:37895 (local address 127.0.0.1:38584)
0620 20:27:57.795584 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.795675 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 20:27:57.796975 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.796602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:37895 (local address 127.0.0.1:38598)
0620 20:27:57.796756 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.796840 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 20:27:57.797300 22776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:37895: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
Jun 20 20:27:57 dist-test-slave-zzwf krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987277, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for kudu/127.31.235.189@KRBTEST.COM
I20260620 20:27:57.807276 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.792758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:45071 (local address 127.0.0.1:41716)
0620 20:27:57.792958 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.792982 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:57.793098 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:57.793641 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:57.793645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:57.794054 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:57.794239 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:57.794247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:57.795788 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:57.795793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:57.796998 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:57.797012 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:57.798086 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:57.798091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:57.798277 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:57.798959 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:27:57.798978 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:57.801302 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:57.804870 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:57.804875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:57.804878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:57.805286 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:57.805718 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:57.805722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:57.805725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:57.805870 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:57.806497 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:57.806501 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:57.806647 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 20:27:57.806763 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 20:27:57.807018 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 20:27:57.807420 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.792867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41716 (local address 127.31.235.189:45071)
0620 20:27:57.793102 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:57.793109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:57.793143 (+ 34us) server_negotiation.cc:407] Connection header received
0620 20:27:57.793224 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:57.793231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:57.793364 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:57.793537 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:57.794506 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:57.795631 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:57.797179 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:57.797908 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:57.798035 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:57.801520 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:57.801527 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:57.801531 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:57.801571 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:57.804730 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:57.805407 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:57.805412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:57.805415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:57.805594 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:57.805982 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:57.805986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:57.805989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:57.806359 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:57.806497 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:57.806903 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 20:27:57.807123 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 20:27:57.808367 22702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:27:57.809437 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.808984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:37895 (local address 127.0.0.1:38608)
0620 20:27:57.809166 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.809258 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 20:27:57.810532 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.810048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:37895 (local address 127.0.0.1:38622)
0620 20:27:57.810257 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.810382 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 20:27:57.810808 22702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:37895: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
I20260620 20:27:57.862105 22702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } attempt: 1
I20260620 20:27:57.863548 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.863002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:37895 (local address 127.0.0.1:38630)
0620 20:27:57.863229 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.863368 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 20:27:57.864583 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.864240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:37895 (local address 127.0.0.1:38640)
0620 20:27:57.864409 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.864478 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 20:27:57.864845 22702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:37895: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
I20260620 20:27:57.888553 22776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } attempt: 3
I20260620 20:27:57.889885 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.889357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:37895 (local address 127.0.0.1:38646)
0620 20:27:57.889619 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.889718 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 20:27:57.890965 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.890545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:37895 (local address 127.0.0.1:38652)
0620 20:27:57.890770 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.890854 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260620 20:27:57.891151 22776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:37895: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
I20260620 20:27:57.933131 22702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } attempt: 2
I20260620 20:27:57.934427 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.933922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:37895 (local address 127.0.0.1:38660)
0620 20:27:57.934172 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.934254 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 20:27:57.935496 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:57.935171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:37895 (local address 127.0.0.1:38664)
0620 20:27:57.935312 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:57.935379 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 20:27:57.935736 22702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:37895: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
W20260620 20:27:58.022308 22784 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:27:58.022636 22784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:27:58.022763 22784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:27:58.022915 22784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:27:58.030652 22784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:27:58.030745 22784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:27:58.030809 22784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:27:58.030855 22784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:27:58.030915 22784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:27:58.034392 22776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } attempt: 4
I20260620 20:27:58.035606 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:58.035090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:37895 (local address 127.0.0.1:38676)
0620 20:27:58.035357 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:58.035445 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 20:27:58.036645 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:58.036319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:37895 (local address 127.0.0.1:38690)
0620 20:27:58.036470 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:58.036545 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 20:27:58.036854 22776 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:37895: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
I20260620 20:27:58.041321 22784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:42077,127.31.235.189:45071,127.31.235.188:37895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.188.keytab
--principal=kudu/127.31.235.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.188:37895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.235.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:27:58.041960 22702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } attempt: 3
I20260620 20:27:58.042876 22784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:27:58.043081 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:58.042599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:37895 (local address 127.0.0.1:38706)
0620 20:27:58.042790 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:58.042869 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 20:27:58.043994 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:58.043662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:37895 (local address 127.0.0.1:38718)
0620 20:27:58.043825 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:58.043897 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 20:27:58.044185 22702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:37895: Network error: Client connection negotiation failed: client connection to 127.31.235.188:37895: connect: Connection refused (error 111)
I20260620 20:27:58.044317 22784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:27:58.056291 22789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:58.056213 22790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:27:58.056253 22792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:27:58.056829 22784 server_base.cc:1034] running on GCE node
I20260620 20:27:58.057689 22784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:27:58.058998 22784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:27:58.060240 22784 hybrid_clock.cc:648] HybridClock initialized: now 1781987278060163 us; error 64 us; skew 500 ppm
I20260620 20:27:58.060665 22784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:27:58 dist-test-slave-zzwf krb5kdc[22622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987278, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:27:58.064021 22784 init.cc:377] Logged in from keytab as kudu/127.31.235.188@KRBTEST.COM (short username kudu)
I20260620 20:27:58.065655 22784 webserver.cc:460] Webserver started at http://127.31.235.188:44343/ using document root <none> and password file <none>
I20260620 20:27:58.066327 22784 fs_manager.cc:359] Metadata directory not provided
I20260620 20:27:58.066457 22784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:27:58.066763 22784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:27:58.069685 22784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-2/data/instance:
uuid: "16e72d8a8e884723bea4038945d267f0"
format_stamp: "Formatted at 2026-06-20 20:27:58 on dist-test-slave-zzwf"
server_key: "602977c55f4e6977ba22c05b75c94521"
server_key_iv: "d53eaa8507770298bd8f682e027a22a8"
server_key_version: "encryptionkey@0"
I20260620 20:27:58.070465 22784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-2/wal/instance:
uuid: "16e72d8a8e884723bea4038945d267f0"
format_stamp: "Formatted at 2026-06-20 20:27:58 on dist-test-slave-zzwf"
server_key: "602977c55f4e6977ba22c05b75c94521"
server_key_iv: "d53eaa8507770298bd8f682e027a22a8"
server_key_version: "encryptionkey@0"
I20260620 20:27:58.075421 22784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:27:58.078965 22799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:27:58.080415 22784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 20:27:58.080600 22784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-2/wal
uuid: "16e72d8a8e884723bea4038945d267f0"
format_stamp: "Formatted at 2026-06-20 20:27:58 on dist-test-slave-zzwf"
server_key: "602977c55f4e6977ba22c05b75c94521"
server_key_iv: "d53eaa8507770298bd8f682e027a22a8"
server_key_version: "encryptionkey@0"
I20260620 20:27:58.080822 22784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:27:58.100445 22784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:27:58.104982 22784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:27:58.105283 22784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:27:58.124346 22784 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.188:37895
I20260620 20:27:58.124439 22850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.188:37895 every 8 connection(s)
I20260620 20:27:58.125763 22784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-2/data/info.pb
I20260620 20:27:58.129434 22851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:27:58.130949 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 22784
I20260620 20:27:58.131135 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781986916608810-32686-0/minicluster-data/master-2/wal/instance
I20260620 20:27:58.131721 32686 external_mini_cluster.cc:1428] Setting key 4a035def7564435d9008ea715fe36f0b
I20260620 20:27:58.139365 22851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42077 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:27:58 dist-test-slave-zzwf krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.188@KRBTEST.COM
Jun 20 20:27:58 dist-test-slave-zzwf krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987278, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.188@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:27:58.155496 22854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:58.133764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38722 (local address 127.31.235.188:37895)
0620 20:27:58.134576 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.134594 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.134637 (+ 43us) server_negotiation.cc:407] Connection header received
0620 20:27:58.136234 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.136256 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.136650 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.137002 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.138700 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.139986 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.141717 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.142813 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.142984 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:58.147379 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:58.147386 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:58.147397 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:58.147432 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:58.150971 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.151643 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.151651 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.151657 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.151800 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.152313 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.152318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.152321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.153051 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:58.153319 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:58.153851 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 20:27:58.154349 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":378,"threads_started":1}
I20260620 20:27:58.157107 22779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:58.142172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.31.235.190:42077)
0620 20:27:58.142345 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.142349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.143114 (+ 765us) server_negotiation.cc:407] Connection header received
0620 20:27:58.143401 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.143405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.143482 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.143561 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.144681 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.145848 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.147235 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.147648 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.147793 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:58.151294 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:58.151299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:58.151302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:58.151337 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:58.154388 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.155056 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.155063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.155069 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.155198 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.155580 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.155586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.155589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.155930 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:58.156096 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:58.156559 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 20:27:58.156786 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 20:27:58.157158 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:58.142185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:42077 (local address 127.0.0.1:40466)
0620 20:27:58.142987 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:58.143030 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:58.143236 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:58.143655 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:58.143665 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:58.144266 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:58.144502 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:58.144510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.146011 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.146024 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:58.147067 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:58.147078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.147838 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.147844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:58.148022 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:58.148609 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:27:58.148625 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:58.151079 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:58.154515 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:58.154527 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:58.154534 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:58.154922 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:58.155317 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:58.155323 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:58.155326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:58.155466 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:58.156059 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:58.156067 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:58.156258 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 20:27:58.156444 (+ 186us) client_negotiation.cc:240] Negotiation successful
0620 20:27:58.156738 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":393,"threads_started":1}
I20260620 20:27:58.158627 22851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:27:58 dist-test-slave-zzwf krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987278, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.188@KRBTEST.COM for kudu/127.31.235.189@KRBTEST.COM
Jun 20 20:27:58 dist-test-slave-zzwf krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:27:58.172086 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:58.159339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:45071 (local address 127.0.0.1:41718)
0620 20:27:58.159498 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:58.159519 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:58.159619 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:58.159955 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:58.159958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:58.160289 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:58.160430 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:58.160438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.161472 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.161480 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:58.162488 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:58.162499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.163302 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.163309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:58.163519 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:58.164012 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:27:58.164050 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:58.166518 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:58.169787 (+ 3269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:58.169794 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:58.169801 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:58.170143 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:58.170509 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:58.170515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:58.170518 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:58.170654 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:58.171147 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:58.171152 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:58.171315 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 20:27:58.171453 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 20:27:58.171717 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 20:27:58.172111 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:58.159413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41718 (local address 127.31.235.189:45071)
0620 20:27:58.159610 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.159617 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.159637 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:27:58.159718 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.159740 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.159822 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.159934 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.160586 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.161324 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.162650 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.163146 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.163299 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:58.166721 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:58.166726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:58.166736 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:58.166779 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:58.169662 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.170261 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.170267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.170271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.170401 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.170757 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.170763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.170767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.171038 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:58.171189 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:58.171549 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 20:27:58.171744 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:27:58.172761 22779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:58.158687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40478 (local address 127.31.235.190:42077)
0620 20:27:58.158905 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.158910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.158927 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:27:58.159004 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.159009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.159085 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.159183 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.160538 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.161230 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.162498 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.162926 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.163052 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:58.167221 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:58.167226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:58.167231 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:58.167269 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:58.170103 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.170754 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.170759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.170764 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.170892 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.171263 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.171270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.171273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.171515 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:58.171842 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:58.172034 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 20:27:58.172237 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 20:27:58.173233 22851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:27:58 dist-test-slave-zzwf krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987278, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.188@KRBTEST.COM for kudu/127.31.235.188@KRBTEST.COM
Jun 20 20:27:58 dist-test-slave-zzwf krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.189@KRBTEST.COM
I20260620 20:27:58.188277 22855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:58.173988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:37895 (local address 127.0.0.1:38726)
0620 20:27:58.174155 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:58.174175 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:58.174288 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:58.174612 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:58.174619 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:58.175125 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:58.175263 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:58.175271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.176585 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.176590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:58.177904 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:58.177914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.178827 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.178831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:58.179030 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:58.179644 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:27:58.179665 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:58.181917 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:58.185548 (+ 3631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:58.185557 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:58.185564 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:58.186102 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:58.186504 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:58.186509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:58.186512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:58.186640 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:58.187178 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:58.187183 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:58.187356 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 20:27:58.187509 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 20:27:58.187832 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 20:27:58.188319 22854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:58.174194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38726 (local address 127.31.235.188:37895)
0620 20:27:58.174353 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.174359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.174376 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:27:58.174421 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.174425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.174493 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.174589 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.175404 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.176459 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.178029 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.178651 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.178786 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:58.182095 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:58.182104 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:58.182112 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:58.182155 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:58.185377 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.186238 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.186245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.186250 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.186379 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.186762 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.186767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.186770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.187058 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:58.187219 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:58.187598 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 20:27:58.187976 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":61}
I20260620 20:27:58.189853 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:58.174443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41724 (local address 127.31.235.189:45071)
0620 20:27:58.174589 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.174602 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.174640 (+ 38us) server_negotiation.cc:407] Connection header received
0620 20:27:58.174813 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.174818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.174884 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.174997 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.176379 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.177538 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.179065 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.179668 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.179807 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:58.184225 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:58.184231 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:58.184235 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:58.184273 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:58.187089 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.187758 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.187763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.187766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.187918 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.188290 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.188296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.188299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.188667 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:58.188959 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:58.189262 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 20:27:58.189520 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 20:27:58.194902 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:27:58.194872 22851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0: Bootstrap starting.
I20260620 20:27:58.198982 22851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:58.200079 22851 log.cc:824] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:58.198567 22677 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "47aac6a2946647c192e482ea0350d716"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40478
I20260620 20:27:58.203135 22851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0: No bootstrap required, opened a new log
I20260620 20:27:58.210873 22851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47aac6a2946647c192e482ea0350d716" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42077 } } peers { permanent_uuid: "33d34284525b406bb2a4da33f7763c29" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } } peers { permanent_uuid: "16e72d8a8e884723bea4038945d267f0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } }
I20260620 20:27:58.211247 22851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16e72d8a8e884723bea4038945d267f0, State: Initialized, Role: FOLLOWER
I20260620 20:27:58.211709 22851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47aac6a2946647c192e482ea0350d716" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42077 } } peers { permanent_uuid: "33d34284525b406bb2a4da33f7763c29" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } } peers { permanent_uuid: "16e72d8a8e884723bea4038945d267f0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } }
I20260620 20:27:58.212731 22857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47aac6a2946647c192e482ea0350d716" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42077 } } peers { permanent_uuid: "33d34284525b406bb2a4da33f7763c29" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } } peers { permanent_uuid: "16e72d8a8e884723bea4038945d267f0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } } }
I20260620 20:27:58.213107 22857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:27:58.213930 22851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:58.221383 22702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } attempt: 4
W20260620 20:27:58.223486 22868 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:27:58.223590 22868 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 20 20:27:58 dist-test-slave-zzwf krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987277, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for kudu/127.31.235.188@KRBTEST.COM
I20260620 20:27:58.221213 22784 master_runner.cc:186] Error getting master registration for 127.31.235.190:42077: OK, instance_id { permanent_uuid: "47aac6a2946647c192e482ea0350d716" instance_seqno: 1781987277076068 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:27:58.234935 22784 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:27:58.240648 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:58.221979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:37895 (local address 127.0.0.1:38732)
0620 20:27:58.225686 (+ 3707us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:58.225719 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:58.225850 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:58.226297 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:58.226319 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:58.226865 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:58.227061 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:58.227073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.228179 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.228187 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:58.229699 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:58.229717 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.230755 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.230761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:58.230968 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:58.231641 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:27:58.231669 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:58.234461 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:58.238176 (+ 3715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:58.238182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:58.238187 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:58.238553 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:58.238954 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:58.238959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:58.238962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:58.239101 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:58.239658 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:58.239663 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:58.239847 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 20:27:58.239983 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 20:27:58.240269 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3558}
I20260620 20:27:58.240653 22854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:58.225660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38732 (local address 127.31.235.188:37895)
0620 20:27:58.225850 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.225856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.225875 (+ 19us) server_negotiation.cc:407] Connection header received
0620 20:27:58.226032 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.226038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.226121 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.226222 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.227202 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.228021 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.229861 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.230609 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.230734 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:58.234658 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:58.234665 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:58.234670 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:58.234706 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:58.238038 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.238680 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.238685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.238689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.238839 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.239215 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.239220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.239224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.239539 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:58.239720 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:58.240095 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 20:27:58.240294 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 20:27:58.245064 22702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716: Bootstrap starting.
I20260620 20:27:58.248523 22702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:58.250125 22702 log.cc:824] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:58.254021 22702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716: No bootstrap required, opened a new log
I20260620 20:27:58.259473 22702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47aac6a2946647c192e482ea0350d716" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42077 } } peers { permanent_uuid: "33d34284525b406bb2a4da33f7763c29" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } } peers { permanent_uuid: "16e72d8a8e884723bea4038945d267f0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } }
I20260620 20:27:58.259773 22702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47aac6a2946647c192e482ea0350d716, State: Initialized, Role: FOLLOWER
I20260620 20:27:58.260143 22702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47aac6a2946647c192e482ea0350d716" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42077 } } peers { permanent_uuid: "33d34284525b406bb2a4da33f7763c29" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } } peers { permanent_uuid: "16e72d8a8e884723bea4038945d267f0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } }
I20260620 20:27:58.261052 22871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47aac6a2946647c192e482ea0350d716" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42077 } } peers { permanent_uuid: "33d34284525b406bb2a4da33f7763c29" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } } peers { permanent_uuid: "16e72d8a8e884723bea4038945d267f0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } } }
I20260620 20:27:58.261451 22871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:27:58.262200 22702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:58.262466 22677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 20:27:58.262740 22677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:58.266914 22677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47aac6a2946647c192e482ea0350d716" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42077 } } peers { permanent_uuid: "33d34284525b406bb2a4da33f7763c29" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } } peers { permanent_uuid: "16e72d8a8e884723bea4038945d267f0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } }
I20260620 20:27:58.269428 22677 leader_election.cc:288] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [CANDIDATE]: Term 1 election: Requested vote from peers 33d34284525b406bb2a4da33f7763c29 (127.31.235.189:45071), 16e72d8a8e884723bea4038945d267f0 (127.31.235.188:37895)
I20260620 20:27:58.270107 22826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47aac6a2946647c192e482ea0350d716" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "16e72d8a8e884723bea4038945d267f0"
I20260620 20:27:58.270711 22826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:58.275341 22826 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47aac6a2946647c192e482ea0350d716 in term 1.
I20260620 20:27:58.276160 22653 leader_election.cc:302] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16e72d8a8e884723bea4038945d267f0, 47aac6a2946647c192e482ea0350d716; no voters:
I20260620 20:27:58.276777 22871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:27:58.278079 22871 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [term 1 LEADER]: Becoming Leader. State: Replica: 47aac6a2946647c192e482ea0350d716, State: Running, Role: LEADER
I20260620 20:27:58.278712 22871 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47aac6a2946647c192e482ea0350d716" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42077 } } peers { permanent_uuid: "33d34284525b406bb2a4da33f7763c29" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } } peers { permanent_uuid: "16e72d8a8e884723bea4038945d267f0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } }
I20260620 20:27:58.284416 22880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47aac6a2946647c192e482ea0350d716. Latest consensus state: current_term: 1 leader_uuid: "47aac6a2946647c192e482ea0350d716" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47aac6a2946647c192e482ea0350d716" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42077 } } peers { permanent_uuid: "33d34284525b406bb2a4da33f7763c29" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } } peers { permanent_uuid: "16e72d8a8e884723bea4038945d267f0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } } }
I20260620 20:27:58.284781 22880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:58.293561 22883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:27:58.304853 22883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 20:27:58.321884 22656 tablet.cc:2369] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:27:58.323446 22893 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 20:27:58.335593 22854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:58.292842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38736 (local address 127.31.235.188:37895)
0620 20:27:58.293047 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.293054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.315782 (+ 22728us) server_negotiation.cc:407] Connection header received
0620 20:27:58.316314 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.316320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.316411 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.316490 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.319876 (+ 3386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.321143 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.323400 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.324224 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.324409 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:58.327854 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:58.327861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:58.327867 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:58.327915 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:58.331996 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.332927 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.332933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.332937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.333109 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.333640 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.333646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.333650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.334005 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:58.334295 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:58.334880 (+ 585us) server_negotiation.cc:299] Negotiation successful
0620 20:27:58.335165 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 20:27:58.336131 22776 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } attempt: 5
Jun 20 20:27:58 dist-test-slave-zzwf krb5kdc[22622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987277, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.189@KRBTEST.COM for kudu/127.31.235.188@KRBTEST.COM
I20260620 20:27:58.349277 22826 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0 [term 1 FOLLOWER]: Refusing update from remote peer 47aac6a2946647c192e482ea0350d716: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:27:58.350921 22880 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16e72d8a8e884723bea4038945d267f0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 20:27:58.357450 22805 tablet.cc:2369] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:27:58.358661 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:58.336695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:37895 (local address 127.0.0.1:38744)
0620 20:27:58.337071 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 20:27:58.337092 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 20:27:58.337198 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:27:58.339636 (+ 2438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:27:58.339642 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:27:58.340193 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:27:58.340424 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:58.340436 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.342230 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.342236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:58.343857 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:27:58.343873 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.345008 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.345015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:27:58.345303 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:58.346133 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:27:58.346156 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:27:58.349470 (+ 3314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:27:58.354258 (+ 4788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:58.354264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:58.354269 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:58.354805 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:58.355434 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:27:58.355441 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:27:58.355446 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:27:58.355657 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:27:58.356372 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:27:58.356379 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:27:58.356553 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 20:27:58.356710 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 20:27:58.357061 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242}
I20260620 20:27:58.360204 22854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:58.337417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38744 (local address 127.31.235.188:37895)
0620 20:27:58.338907 (+ 1490us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.338913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.338933 (+ 20us) server_negotiation.cc:407] Connection header received
0620 20:27:58.338985 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.338990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.339064 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.340608 (+ 1544us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.340770 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.341973 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.344059 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.344824 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.345661 (+ 837us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:58.349741 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:58.349750 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:58.349754 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:58.349801 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:58.354076 (+ 4275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.355019 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.355026 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.355031 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.355203 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.355816 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.355822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.355826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.356206 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:58.359343 (+ 3137us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:58.359484 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 20:27:58.359747 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1369,"thread_start_us":4641,"threads_started":1}
I20260620 20:27:58.330758 22635 master_runner.cc:186] Error getting master registration for 127.31.235.189:45071: OK, instance_id { permanent_uuid: "33d34284525b406bb2a4da33f7763c29" instance_seqno: 1781987277603803 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:27:58.365650 22635 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:27:58.367177 22776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33d34284525b406bb2a4da33f7763c29: Bootstrap starting.
I20260620 20:27:58.367632 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:58.289620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.31.235.189:45071)
0620 20:27:58.289841 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.289850 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.293506 (+ 3656us) server_negotiation.cc:407] Connection header received
0620 20:27:58.294333 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.294339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.294428 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.294515 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.304444 (+ 9929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.305587 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.314748 (+ 9161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.315645 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.315761 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:58.322693 (+ 6932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:58.322700 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:58.322704 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:58.322745 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:58.326483 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.348181 (+ 21698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.348187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.348190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.348374 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.358675 (+ 10301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.358682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.358685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.359048 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:58.359317 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:58.366917 (+ 7600us) server_negotiation.cc:299] Negotiation successful
0620 20:27:58.367243 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 20:27:58.372167 22776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33d34284525b406bb2a4da33f7763c29: Neither blocks nor log segments found. Creating new log.
I20260620 20:27:58.373741 22776 log.cc:824] T 00000000000000000000000000000000 P 33d34284525b406bb2a4da33f7763c29: Log is configured to *not* fsync() on all Append() calls
I20260620 20:27:58.378883 22779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:58.289022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40486 (local address 127.31.235.190:42077)
0620 20:27:58.289241 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.289250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.300336 (+ 11086us) server_negotiation.cc:407] Connection header received
0620 20:27:58.301034 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.301040 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.301132 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.301209 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.310959 (+ 9750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.312171 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.318515 (+ 6344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:58.325443 (+ 6928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:58.325613 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:58.361113 (+ 35500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:27:58.361122 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:27:58.361127 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:27:58.361177 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:27:58.365215 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.373189 (+ 7974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.373196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.373200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.373401 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:27:58.374942 (+ 1541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:27:58.374969 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:27:58.374973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:27:58.375394 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:27:58.375698 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:58.378221 (+ 2523us) server_negotiation.cc:299] Negotiation successful
0620 20:27:58.378509 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:27:58.384017 22776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33d34284525b406bb2a4da33f7763c29: No bootstrap required, opened a new log
I20260620 20:27:58.396255 22857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47aac6a2946647c192e482ea0350d716. Latest consensus state: current_term: 1 leader_uuid: "47aac6a2946647c192e482ea0350d716" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47aac6a2946647c192e482ea0350d716" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42077 } } peers { permanent_uuid: "33d34284525b406bb2a4da33f7763c29" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } } peers { permanent_uuid: "16e72d8a8e884723bea4038945d267f0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } } }
I20260620 20:27:58.396591 22857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:27:58.396276 22776 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33d34284525b406bb2a4da33f7763c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47aac6a2946647c192e482ea0350d716" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42077 } } peers { permanent_uuid: "33d34284525b406bb2a4da33f7763c29" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } } peers { permanent_uuid: "16e72d8a8e884723bea4038945d267f0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } }
I20260620 20:27:58.396888 22776 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33d34284525b406bb2a4da33f7763c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d34284525b406bb2a4da33f7763c29, State: Initialized, Role: FOLLOWER
I20260620 20:27:58.397469 22776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33d34284525b406bb2a4da33f7763c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47aac6a2946647c192e482ea0350d716" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42077 } } peers { permanent_uuid: "33d34284525b406bb2a4da33f7763c29" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } } peers { permanent_uuid: "16e72d8a8e884723bea4038945d267f0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } }
I20260620 20:27:58.404512 22899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d34284525b406bb2a4da33f7763c29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47aac6a2946647c192e482ea0350d716" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42077 } } peers { permanent_uuid: "33d34284525b406bb2a4da33f7763c29" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } } peers { permanent_uuid: "16e72d8a8e884723bea4038945d267f0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } } }
I20260620 20:27:58.406809 22899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d34284525b406bb2a4da33f7763c29 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:27:58.406677 22776 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33d34284525b406bb2a4da33f7763c29 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:27:58.406737 22751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47aac6a2946647c192e482ea0350d716" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "33d34284525b406bb2a4da33f7763c29"
I20260620 20:27:58.407064 22750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33d34284525b406bb2a4da33f7763c29 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:27:58.410805 22750 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33d34284525b406bb2a4da33f7763c29 [term 1 FOLLOWER]: Refusing update from remote peer 47aac6a2946647c192e482ea0350d716: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:27:58.412150 22896 mvcc.cc:205] Tried to move back new op lower bound from 7299019892100378624 to 7299019891835428864. Current Snapshot: MvccSnapshot[applied={T|T < 7299019892100378624 or (T in {7299019892100378624})}]
I20260620 20:27:58.414567 22857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47aac6a2946647c192e482ea0350d716" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47aac6a2946647c192e482ea0350d716" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42077 } } peers { permanent_uuid: "33d34284525b406bb2a4da33f7763c29" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } } peers { permanent_uuid: "16e72d8a8e884723bea4038945d267f0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } } }
I20260620 20:27:58.414789 22857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:27:58.414896 22751 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 33d34284525b406bb2a4da33f7763c29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47aac6a2946647c192e482ea0350d716 in term 1.
I20260620 20:27:58.417258 22880 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d34284525b406bb2a4da33f7763c29" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:27:58.420797 22880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47aac6a2946647c192e482ea0350d716" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47aac6a2946647c192e482ea0350d716" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42077 } } peers { permanent_uuid: "33d34284525b406bb2a4da33f7763c29" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } } peers { permanent_uuid: "16e72d8a8e884723bea4038945d267f0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } } }
I20260620 20:27:58.421046 22880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:58.426210 22883 catalog_manager.cc:1347] Generated new cluster ID: c8d18bb6df9b47e09b2b3c2193f2a9f9
I20260620 20:27:58.426359 22883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:27:58.444236 22883 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:27:58.446085 22883 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:27:58.469691 22880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47aac6a2946647c192e482ea0350d716" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47aac6a2946647c192e482ea0350d716" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42077 } } peers { permanent_uuid: "33d34284525b406bb2a4da33f7763c29" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } } peers { permanent_uuid: "16e72d8a8e884723bea4038945d267f0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } } }
I20260620 20:27:58.469985 22880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716 [sys.catalog]: This master's current role is: LEADER
I20260620 20:27:58.477768 22883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47aac6a2946647c192e482ea0350d716: Generated new TSK 0
I20260620 20:27:58.478910 22883 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:27:58.482131 22916 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 20:27:58.503041 22899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d34284525b406bb2a4da33f7763c29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47aac6a2946647c192e482ea0350d716. Latest consensus state: current_term: 1 leader_uuid: "47aac6a2946647c192e482ea0350d716" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47aac6a2946647c192e482ea0350d716" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42077 } } peers { permanent_uuid: "33d34284525b406bb2a4da33f7763c29" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } } peers { permanent_uuid: "16e72d8a8e884723bea4038945d267f0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } } }
I20260620 20:27:58.503358 22899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d34284525b406bb2a4da33f7763c29 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:27:58.504647 22911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d34284525b406bb2a4da33f7763c29 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47aac6a2946647c192e482ea0350d716" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47aac6a2946647c192e482ea0350d716" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 42077 } } peers { permanent_uuid: "33d34284525b406bb2a4da33f7763c29" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 45071 } } peers { permanent_uuid: "16e72d8a8e884723bea4038945d267f0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 37895 } } }
I20260620 20:27:58.505049 22911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d34284525b406bb2a4da33f7763c29 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 20:27:58.570964 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:27:58.563216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.31.235.189:45071)
0620 20:27:58.563406 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.563413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.567662 (+ 4249us) server_negotiation.cc:407] Connection header received
0620 20:27:58.569287 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.569295 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.569386 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.569467 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.570700 (+ 1233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41738: BlockingRecv error: recv got EOF from 127.0.0.1:41738 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 20:27:58.573115 22779 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:27:58.563862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40502 (local address 127.31.235.190:42077)
0620 20:27:58.564056 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.564062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.569016 (+ 4954us) server_negotiation.cc:407] Connection header received
0620 20:27:58.571610 (+ 2594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.571616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.571697 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.571764 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.572854 (+ 1090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40502: BlockingRecv error: recv got EOF from 127.0.0.1:40502 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 20:27:58.575668 22854 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:27:58.567814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.31.235.188:37895)
0620 20:27:58.567981 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.567986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.573881 (+ 5895us) server_negotiation.cc:407] Connection header received
0620 20:27:58.574362 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.574368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.574449 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.574516 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.575426 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38760: BlockingRecv error: recv got EOF from 127.0.0.1:38760 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 20:27:58.583711 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:27:58.581671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41740 (local address 127.31.235.189:45071)
0620 20:27:58.581858 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.581864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.582299 (+ 435us) server_negotiation.cc:407] Connection header received
0620 20:27:58.582497 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.582503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.582625 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.582737 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.583455 (+ 718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41740: BlockingRecv error: recv got EOF from 127.0.0.1:41740 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 20:27:58.595592 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:27:58.591819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41744 (local address 127.31.235.189:45071)
0620 20:27:58.591992 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.591998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.594059 (+ 2061us) server_negotiation.cc:407] Connection header received
0620 20:27:58.594262 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.594268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.594365 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.594479 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.595355 (+ 876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41744: BlockingRecv error: recv got EOF from 127.0.0.1:41744 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 20:27:58.607558 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:27:58.600863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.31.235.189:45071)
0620 20:27:58.601026 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.601032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.606133 (+ 5101us) server_negotiation.cc:407] Connection header received
0620 20:27:58.606337 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.606342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.606431 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.606539 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.607318 (+ 779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41748: BlockingRecv error: recv got EOF from 127.0.0.1:41748 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 20:27:58.647188 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:27:58.641957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.31.235.189:45071)
0620 20:27:58.642176 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.642183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.645277 (+ 3094us) server_negotiation.cc:407] Connection header received
0620 20:27:58.645640 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.645647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.645745 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.645826 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.646952 (+ 1126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41756: BlockingRecv error: recv got EOF from 127.0.0.1:41756 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260620 20:27:58.662036 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:27:58.659615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41766 (local address 127.31.235.189:45071)
0620 20:27:58.659796 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.659802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.660324 (+ 522us) server_negotiation.cc:407] Connection header received
0620 20:27:58.660610 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.660617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.660725 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.660801 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.661793 (+ 992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41766: BlockingRecv error: recv got EOF from 127.0.0.1:41766 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 20:27:58.685439 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:27:58.682679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41772 (local address 127.31.235.189:45071)
0620 20:27:58.682899 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.682906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.683409 (+ 503us) server_negotiation.cc:407] Connection header received
0620 20:27:58.683804 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.683811 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.683940 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.684019 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.685158 (+ 1139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41772: BlockingRecv error: recv got EOF from 127.0.0.1:41772 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 20:27:58.726178 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:27:58.724035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41782 (local address 127.31.235.189:45071)
0620 20:27:58.724235 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.724241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.724288 (+ 47us) server_negotiation.cc:407] Connection header received
0620 20:27:58.724667 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.724673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.724780 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.724928 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.725891 (+ 963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41782: BlockingRecv error: recv got EOF from 127.0.0.1:41782 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 20:27:58.796243 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:27:58.794150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41798 (local address 127.31.235.189:45071)
0620 20:27:58.794429 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.794441 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.794780 (+ 339us) server_negotiation.cc:407] Connection header received
0620 20:27:58.794993 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.794997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.795073 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.795160 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.796029 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41798: BlockingRecv error: recv got EOF from 127.0.0.1:41798 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 20:27:58.929950 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:27:58.928028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.31.235.189:45071)
0620 20:27:58.928264 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:58.928271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:58.928477 (+ 206us) server_negotiation.cc:407] Connection header received
0620 20:27:58.928655 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:58.928659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:58.928745 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:58.928841 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:58.929679 (+ 838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41806: BlockingRecv error: recv got EOF from 127.0.0.1:41806 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 20:27:59.191324 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:27:59.189548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41814 (local address 127.31.235.189:45071)
0620 20:27:59.189753 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:59.189759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:59.189887 (+ 128us) server_negotiation.cc:407] Connection header received
0620 20:27:59.190132 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:59.190136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:59.190207 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:59.190324 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:27:59.191084 (+ 760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41814: BlockingRecv error: recv got EOF from 127.0.0.1:41814 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:27:59.225538 22868 catalog_manager.cc:1259] Loaded cluster ID: c8d18bb6df9b47e09b2b3c2193f2a9f9
I20260620 20:27:59.225649 22868 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0: loading cluster ID for follower catalog manager: success
I20260620 20:27:59.227975 22868 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0: acquiring CA information for follower catalog manager: success
I20260620 20:27:59.229768 22868 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 16e72d8a8e884723bea4038945d267f0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:27:59.486559 22916 catalog_manager.cc:1259] Loaded cluster ID: c8d18bb6df9b47e09b2b3c2193f2a9f9
I20260620 20:27:59.486755 22916 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 33d34284525b406bb2a4da33f7763c29: loading cluster ID for follower catalog manager: success
I20260620 20:27:59.489377 22916 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 33d34284525b406bb2a4da33f7763c29: acquiring CA information for follower catalog manager: success
I20260620 20:27:59.491860 22916 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 33d34284525b406bb2a4da33f7763c29: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:27:59.715325 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:59.707549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.31.235.189:45071)
0620 20:27:59.708148 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:59.708155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:59.708473 (+ 318us) server_negotiation.cc:407] Connection header received
0620 20:27:59.708720 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:59.708725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:59.708921 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:59.709070 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:27:59.709618 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:59.710834 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:59.712269 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:59.712960 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:59.713093 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:59.713669 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:27:59.714434 (+ 765us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:27:59.714603 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:59.714769 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 20:27:59.714875 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":274,"threads_started":1}
I20260620 20:27:59.734119 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:59.728199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38762 (local address 127.31.235.188:37895)
0620 20:27:59.728916 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:59.728924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:59.728946 (+ 22us) server_negotiation.cc:407] Connection header received
0620 20:27:59.729058 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:59.729062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:59.729226 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:59.729423 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:27:59.729870 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:59.730605 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:59.731883 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:59.732300 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:59.732432 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:59.732934 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:27:59.733432 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:27:59.733651 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:59.733704 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 20:27:59.733798 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":474,"threads_started":1}
I20260620 20:27:59.757079 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:59.740771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41828 (local address 127.31.235.189:45071)
0620 20:27:59.740946 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:59.740952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:59.747830 (+ 6878us) server_negotiation.cc:407] Connection header received
0620 20:27:59.748766 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:59.748776 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:59.748901 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:59.749023 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:27:59.750006 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:59.751570 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:59.754398 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:59.755373 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:59.755507 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:59.756196 (+ 689us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:27:59.756349 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:27:59.756425 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:59.756644 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 20:27:59.756738 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:27:59.763984 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:59.740698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40506 (local address 127.31.235.190:42077)
0620 20:27:59.756183 (+ 15485us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:59.756189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:59.756207 (+ 18us) server_negotiation.cc:407] Connection header received
0620 20:27:59.756320 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:59.756325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:59.756551 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:59.757429 (+ 878us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:27:59.757610 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:59.758737 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:59.760866 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:59.761461 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:59.762120 (+ 659us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:59.762300 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:27:59.762852 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:27:59.763480 (+ 628us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:59.763534 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 20:27:59.763657 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15331,"thread_start_us":16007,"threads_started":1}
I20260620 20:27:59.764046 22989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:27:59.748529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38768 (local address 127.31.235.188:37895)
0620 20:27:59.748710 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 20:27:59.748716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:27:59.757876 (+ 9160us) server_negotiation.cc:407] Connection header received
0620 20:27:59.758118 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:27:59.758125 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:27:59.758233 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:27:59.758295 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:27:59.758794 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:59.759875 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:59.762017 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:27:59.762607 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:27:59.762706 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:27:59.763312 (+ 606us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:27:59.763441 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:27:59.763503 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 20:27:59.763727 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 20:27:59.763803 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 20:27:59.769150 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 22635
I20260620 20:27:59.770285 22697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:27:59.886245 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 22635
W20260620 20:27:59.906898 22800 connection.cc:535] server connection from 127.0.0.1:38732 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 20:27:59.907563 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 22709
I20260620 20:27:59.908811 22771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:28:00.020164 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 22709
I20260620 20:28:00.040230 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 22784
I20260620 20:28:00.041453 22846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:28:00.148741 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 22784
2026-06-20T20:28:00Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4897 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 20:28:00 dist-test-slave-zzwf krb5kdc[23003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 20:28:00 dist-test-slave-zzwf krb5kdc[23003](info): set up 2 sockets
Jun 20 20:28:00 dist-test-slave-zzwf krb5kdc[23003](info): commencing operation
krb5kdc: starting...
W20260620 20:28:01.282400 32686 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/test-user.keytab.
Jun 20 20:28:01 dist-test-slave-zzwf krb5kdc[23003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T20:28:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T20:28:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.235.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
Entry for principal kudu/127.31.235.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab.
I20260620 20:28:01.537031 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:46263
--webserver_interface=127.31.235.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:43849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:46263,127.31.235.189:39573,127.31.235.188:39757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 20:28:01.875494 23016 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:28:01.875926 23016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:28:01.876057 23016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:28:01.876195 23016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:28:01.884451 23016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:28:01.884644 23016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:28:01.884780 23016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:28:01.884891 23016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:28:01.885030 23016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:28:01.896256 23016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:46263,127.31.235.189:39573,127.31.235.188:39757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.190.keytab
--principal=kudu/127.31.235.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.190:46263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.235.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:28:01.897853 23016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:28:01.899291 23016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:28:01.911123 23022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:28:01.911291 23021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:28:01.911151 23024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:28:01.912088 23016 server_base.cc:1034] running on GCE node
I20260620 20:28:01.912937 23016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:28:01.914472 23016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:28:01.915740 23016 hybrid_clock.cc:648] HybridClock initialized: now 1781987281915683 us; error 48 us; skew 500 ppm
I20260620 20:28:01.916108 23016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:28:01 dist-test-slave-zzwf krb5kdc[23003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987281, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:28:01.919421 23016 init.cc:377] Logged in from keytab as kudu/127.31.235.190@KRBTEST.COM (short username kudu)
I20260620 20:28:01.921087 23016 webserver.cc:460] Webserver started at http://127.31.235.190:34681/ using document root <none> and password file <none>
I20260620 20:28:01.921898 23016 fs_manager.cc:359] Metadata directory not provided
I20260620 20:28:01.921998 23016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:28:01.922255 23016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:28:01.925195 23016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-0/data/instance:
uuid: "2571456d35854c32b00534564be74ab0"
format_stamp: "Formatted at 2026-06-20 20:28:01 on dist-test-slave-zzwf"
server_key: "9e06450494d27e107e8231a33eb3b471"
server_key_iv: "3d6b9e5d1fd98e16cf97995d4e8411bf"
server_key_version: "encryptionkey@0"
I20260620 20:28:01.926084 23016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-0/wal/instance:
uuid: "2571456d35854c32b00534564be74ab0"
format_stamp: "Formatted at 2026-06-20 20:28:01 on dist-test-slave-zzwf"
server_key: "9e06450494d27e107e8231a33eb3b471"
server_key_iv: "3d6b9e5d1fd98e16cf97995d4e8411bf"
server_key_version: "encryptionkey@0"
I20260620 20:28:01.931627 23016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 20:28:01.935361 23031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:28:01.937134 23016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 20:28:01.937314 23016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-0/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-0/wal
uuid: "2571456d35854c32b00534564be74ab0"
format_stamp: "Formatted at 2026-06-20 20:28:01 on dist-test-slave-zzwf"
server_key: "9e06450494d27e107e8231a33eb3b471"
server_key_iv: "3d6b9e5d1fd98e16cf97995d4e8411bf"
server_key_version: "encryptionkey@0"
I20260620 20:28:01.937628 23016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:28:01.960244 23016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:28:01.964884 23016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:28:01.965119 23016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:28:01.985230 23016 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.190:46263
I20260620 20:28:01.985250 23082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.190:46263 every 8 connection(s)
I20260620 20:28:01.986732 23016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-0/data/info.pb
I20260620 20:28:01.990588 23083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:28:01.992377 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 23016
I20260620 20:28:01.992559 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-0/wal/instance
I20260620 20:28:01.993146 32686 external_mini_cluster.cc:1428] Setting key b42c6f2ebef8543a54a81b8914999e5b
I20260620 20:28:02.000957 23083 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46263 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:28:02 dist-test-slave-zzwf krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987281, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.235.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.189@KRBTEST.COM" created.
I20260620 20:28:02.027547 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.007104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41714 (local address 127.31.235.190:46263)
0620 20:28:02.007939 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:02.007947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:02.007998 (+ 51us) server_negotiation.cc:407] Connection header received
0620 20:28:02.009220 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:02.009226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:02.009484 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:02.009751 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:02.011074 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:02.012266 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:02.013763 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:02.014333 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:02.014554 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:02.018678 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:28:02.018687 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:28:02.018700 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:28:02.018731 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:28:02.022783 (+ 4052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:02.023600 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:02.023607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:02.023610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:02.023771 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:02.024395 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:02.024402 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:02.024405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:02.025100 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:28:02.025311 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 20:28:02.026081 (+ 770us) server_negotiation.cc:299] Negotiation successful
0620 20:28:02.026357 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":418,"threads_started":1}
I20260620 20:28:02.027576 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.005655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46263 (local address 127.0.0.1:41714)
0620 20:28:02.006545 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.006588 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 20:28:02.008806 (+ 2218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:28:02.009766 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:28:02.009773 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:28:02.010498 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:28:02.010835 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:28:02.010853 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:28:02.012433 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:28:02.012438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:28:02.013615 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:28:02.013628 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:28:02.014591 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:28:02.014598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:28:02.014800 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:02.015536 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:28:02.015561 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:28:02.018419 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:28:02.022975 (+ 4556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:28:02.022987 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:28:02.023002 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:28:02.023447 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:28:02.023913 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:28:02.023917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:28:02.023921 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:28:02.024116 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:28:02.025283 (+ 1167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:28:02.025296 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:28:02.025600 (+ 304us) client_negotiation.cc:769] Sending connection context
0620 20:28:02.025903 (+ 303us) client_negotiation.cc:240] Negotiation successful
0620 20:28:02.026589 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":123,"thread_start_us":368,"threads_started":1}
I20260620 20:28:02.030993 23083 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:28:02.032301 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.031822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:39573 (local address 127.0.0.1:52320)
0620 20:28:02.032033 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.032151 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:39573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260620 20:28:02.032765 23032 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.235.189:39573: connect: Connection refused (error 111)
I20260620 20:28:02.034483 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.034100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:39573 (local address 127.0.0.1:52326)
0620 20:28:02.034281 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.034355 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:39573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 20:28:02.034695 23083 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.189:39573: Network error: Client connection negotiation failed: client connection to 127.31.235.189:39573: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.189.keytab.
Entry for principal kudu/127.31.235.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.189.keytab.
I20260620 20:28:02.060391 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.189:39573
--webserver_interface=127.31.235.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.189.keytab
--principal=kudu/127.31.235.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:43849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:46263,127.31.235.189:39573,127.31.235.188:39757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:28:02.084177 23083 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } attempt: 1
I20260620 20:28:02.085379 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.084919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:39573 (local address 127.0.0.1:52334)
0620 20:28:02.085123 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.085242 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:39573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 20:28:02.086396 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.086054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:39573 (local address 127.0.0.1:52338)
0620 20:28:02.086194 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.086267 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:39573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 20:28:02.086588 23083 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.189:39573: Network error: Client connection negotiation failed: client connection to 127.31.235.189:39573: connect: Connection refused (error 111)
I20260620 20:28:02.154855 23083 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } attempt: 2
I20260620 20:28:02.156047 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.155557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:39573 (local address 127.0.0.1:52354)
0620 20:28:02.155812 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.155911 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:39573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 20:28:02.157042 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.156695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:39573 (local address 127.0.0.1:52362)
0620 20:28:02.156836 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.156922 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:39573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 20:28:02.157222 23083 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.189:39573: Network error: Client connection negotiation failed: client connection to 127.31.235.189:39573: connect: Connection refused (error 111)
I20260620 20:28:02.248523 23083 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } attempt: 3
I20260620 20:28:02.249976 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.249361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:39573 (local address 127.0.0.1:52366)
0620 20:28:02.249661 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.249782 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:39573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 20:28:02.251331 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.250811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:39573 (local address 127.0.0.1:52380)
0620 20:28:02.251028 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.251124 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:39573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260620 20:28:02.251581 23083 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.189:39573: Network error: Client connection negotiation failed: client connection to 127.31.235.189:39573: connect: Connection refused (error 111)
I20260620 20:28:02.394861 23083 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } attempt: 4
I20260620 20:28:02.396073 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.395525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:39573 (local address 127.0.0.1:52382)
0620 20:28:02.395833 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.395926 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:39573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260620 20:28:02.397053 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.396707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:39573 (local address 127.0.0.1:52388)
0620 20:28:02.396856 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.396932 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.189:39573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 20:28:02.397280 23083 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.189:39573: Network error: Client connection negotiation failed: client connection to 127.31.235.189:39573: connect: Connection refused (error 111)
W20260620 20:28:02.399140 23090 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:28:02.399489 23090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:28:02.399581 23090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:28:02.399727 23090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:28:02.407035 23090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:28:02.407115 23090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:28:02.407208 23090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:28:02.407300 23090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:28:02.407420 23090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:28:02.417245 23090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:46263,127.31.235.189:39573,127.31.235.188:39757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.189.keytab
--principal=kudu/127.31.235.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.189:39573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.235.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:28:02.418535 23090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:28:02.419831 23090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 20:28:02.431734 23095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:28:02.431867 23098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:28:02.432009 23096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:28:02.432780 23090 server_base.cc:1034] running on GCE node
I20260620 20:28:02.433743 23090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:28:02.435221 23090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:28:02.436499 23090 hybrid_clock.cc:648] HybridClock initialized: now 1781987282436408 us; error 56 us; skew 500 ppm
I20260620 20:28:02.436868 23090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 20:28:02 dist-test-slave-zzwf krb5kdc[23003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987282, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:28:02.440333 23090 init.cc:377] Logged in from keytab as kudu/127.31.235.189@KRBTEST.COM (short username kudu)
I20260620 20:28:02.441936 23090 webserver.cc:460] Webserver started at http://127.31.235.189:33705/ using document root <none> and password file <none>
I20260620 20:28:02.442687 23090 fs_manager.cc:359] Metadata directory not provided
I20260620 20:28:02.442829 23090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:28:02.443182 23090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:28:02.446497 23090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-1/data/instance:
uuid: "95aea5a6bc5441d8aa61deacfe11141d"
format_stamp: "Formatted at 2026-06-20 20:28:02 on dist-test-slave-zzwf"
server_key: "3745d00bb127c2e34edeb10ae4daa315"
server_key_iv: "8630ec6ac4138c209e2665353589d2aa"
server_key_version: "encryptionkey@0"
I20260620 20:28:02.447376 23090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-1/wal/instance:
uuid: "95aea5a6bc5441d8aa61deacfe11141d"
format_stamp: "Formatted at 2026-06-20 20:28:02 on dist-test-slave-zzwf"
server_key: "3745d00bb127c2e34edeb10ae4daa315"
server_key_iv: "8630ec6ac4138c209e2665353589d2aa"
server_key_version: "encryptionkey@0"
I20260620 20:28:02.452757 23090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 20:28:02.456609 23105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:28:02.458117 23090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:28:02.458321 23090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-1/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-1/wal
uuid: "95aea5a6bc5441d8aa61deacfe11141d"
format_stamp: "Formatted at 2026-06-20 20:28:02 on dist-test-slave-zzwf"
server_key: "3745d00bb127c2e34edeb10ae4daa315"
server_key_iv: "8630ec6ac4138c209e2665353589d2aa"
server_key_version: "encryptionkey@0"
I20260620 20:28:02.458563 23090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:28:02.478925 23090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:28:02.483659 23090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:28:02.483983 23090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:28:02.504413 23090 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.189:39573
I20260620 20:28:02.504444 23156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.189:39573 every 8 connection(s)
I20260620 20:28:02.505930 23090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-1/data/info.pb
I20260620 20:28:02.507156 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 23090
I20260620 20:28:02.507347 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-1/wal/instance
I20260620 20:28:02.507925 32686 external_mini_cluster.cc:1428] Setting key 1d6ffa219b0de8c964f49b20cef0893f
I20260620 20:28:02.510676 23157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:28:02.520048 23157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46263 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.235.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.235.188@KRBTEST.COM" created.
Jun 20 20:28:02 dist-test-slave-zzwf krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987282, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.189@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:28:02.546432 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.523278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41716 (local address 127.31.235.190:46263)
0620 20:28:02.523509 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:02.523516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:02.524757 (+ 1241us) server_negotiation.cc:407] Connection header received
0620 20:28:02.526855 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:02.526864 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:02.526969 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:02.527087 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:02.528711 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:02.529931 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:02.531804 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:02.532569 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:02.532724 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:02.538869 (+ 6145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:28:02.538880 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:28:02.538889 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:28:02.538943 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:28:02.542932 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:02.543673 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:02.543682 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:02.543689 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:02.543851 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:02.544267 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:02.544275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:02.544282 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:02.544723 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:28:02.544890 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 20:28:02.545653 (+ 763us) server_negotiation.cc:299] Negotiation successful
0620 20:28:02.545946 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 20:28:02.547056 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.523708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46263 (local address 127.0.0.1:41716)
0620 20:28:02.524573 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.524627 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 20:28:02.526573 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:28:02.527338 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:28:02.527347 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:28:02.527914 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:28:02.528519 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:28:02.528538 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:28:02.530142 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:28:02.530147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:28:02.531640 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:28:02.531654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:28:02.532830 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:28:02.532836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:28:02.533075 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:02.533881 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:28:02.533905 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:28:02.538593 (+ 4688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:28:02.543082 (+ 4489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:28:02.543089 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:28:02.543104 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:28:02.543545 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:28:02.543966 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:28:02.543970 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:28:02.543972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:28:02.544139 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:28:02.544850 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:28:02.544856 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:28:02.545216 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 20:28:02.545454 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 20:28:02.545920 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":361,"threads_started":1}
I20260620 20:28:02.548926 23157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:28:02 dist-test-slave-zzwf krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987282, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.189@KRBTEST.COM for kudu/127.31.235.189@KRBTEST.COM
I20260620 20:28:02.568701 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.549736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:39573 (local address 127.0.0.1:52398)
0620 20:28:02.549947 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.549969 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:28:02.550076 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:28:02.551022 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:28:02.551030 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:28:02.551519 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:28:02.551730 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:28:02.551744 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:28:02.553096 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:28:02.553104 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:28:02.554663 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:28:02.554679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:28:02.555464 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:28:02.555473 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:28:02.555692 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:02.559828 (+ 4136us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:28:02.559853 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:28:02.562737 (+ 2884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:28:02.566300 (+ 3563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:28:02.566304 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:28:02.566307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:28:02.566653 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:28:02.567110 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:28:02.567113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:28:02.567115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:28:02.567238 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:28:02.567923 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:28:02.567926 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:28:02.568036 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 20:28:02.568138 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 20:28:02.568408 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 20:28:02.568912 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.549946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52398 (local address 127.31.235.189:39573)
0620 20:28:02.550612 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:02.550617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:02.550646 (+ 29us) server_negotiation.cc:407] Connection header received
0620 20:28:02.550690 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:02.550696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:02.550856 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:02.550972 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:02.551904 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:02.552953 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:02.554794 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:02.555312 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:02.555433 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:02.562974 (+ 7541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:28:02.562986 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:28:02.563000 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:28:02.563030 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:28:02.566154 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:02.566793 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:02.566799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:02.566802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:02.566981 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:02.567335 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:02.567338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:02.567340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:02.567802 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:28:02.567959 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 20:28:02.568327 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 20:28:02.568539 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":387,"threads_started":1}
I20260620 20:28:02.571350 23157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.235.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.188.keytab.
Entry for principal kudu/127.31.235.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.188.keytab.
I20260620 20:28:02.572700 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.572211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:39757 (local address 127.0.0.1:51658)
0620 20:28:02.572407 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.572563 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 20:28:02.572984 23107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
I20260620 20:28:02.573565 32686 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
/tmp/dist-test-taskgJsYo5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.188:39757
--webserver_interface=127.31.235.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.188.keytab
--principal=kudu/127.31.235.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.235.148:43849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.235.190:46263,127.31.235.189:39573,127.31.235.188:39757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 20:28:02.574524 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.574182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:39757 (local address 127.0.0.1:51672)
0620 20:28:02.574351 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.574431 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 20:28:02.574754 23157 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:39757: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
I20260620 20:28:02.624074 23157 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } attempt: 1
I20260620 20:28:02.625309 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.624860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:39757 (local address 127.0.0.1:51682)
0620 20:28:02.625056 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.625159 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 20:28:02.626569 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.626119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:39757 (local address 127.0.0.1:51694)
0620 20:28:02.626314 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.626429 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 20:28:02.626809 23157 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:39757: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
I20260620 20:28:02.695061 23157 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } attempt: 2
I20260620 20:28:02.696321 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.695796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:39757 (local address 127.0.0.1:51710)
0620 20:28:02.696012 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.696121 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 20:28:02.696535 23083 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } attempt: 5
I20260620 20:28:02.697530 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.697046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:39757 (local address 127.0.0.1:51716)
0620 20:28:02.697247 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.697331 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260620 20:28:02.697809 23157 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:39757: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
Jun 20 20:28:02 dist-test-slave-zzwf krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987281, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for kudu/127.31.235.189@KRBTEST.COM
I20260620 20:28:02.714962 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.697126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:39573 (local address 127.0.0.1:52414)
0620 20:28:02.697370 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.697386 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 20:28:02.697546 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:28:02.698012 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:28:02.698017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:28:02.698501 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:28:02.698700 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:28:02.698712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:28:02.699699 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:28:02.699705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:28:02.701080 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:28:02.701095 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:28:02.701971 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:28:02.701980 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:28:02.702281 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:02.702948 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:28:02.702971 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:28:02.705659 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:28:02.711160 (+ 5501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:28:02.711166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:28:02.711170 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:28:02.712048 (+ 878us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:28:02.712607 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:28:02.712615 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:28:02.712620 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:28:02.712800 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:28:02.713685 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:28:02.713690 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:28:02.713864 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 20:28:02.714061 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 20:28:02.714570 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260620 20:28:02.715332 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.697205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52414 (local address 127.31.235.189:39573)
0620 20:28:02.697362 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:02.697368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:02.697468 (+ 100us) server_negotiation.cc:407] Connection header received
0620 20:28:02.697719 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:02.697727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:02.697852 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:02.697974 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:02.698858 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:02.699559 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:02.701254 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:02.701807 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:02.701916 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:02.706220 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:28:02.706230 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:28:02.706236 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:28:02.706289 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:28:02.710930 (+ 4641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:02.712273 (+ 1343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:02.712278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:02.712282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:02.712476 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:02.712931 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:02.712938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:02.712943 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:02.713555 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:28:02.713723 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 20:28:02.714301 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 20:28:02.714832 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:28:02.716922 23083 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:28:02.718317 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.717933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:39757 (local address 127.0.0.1:51732)
0620 20:28:02.718103 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.718191 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 20:28:02.719436 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.719090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:39757 (local address 127.0.0.1:51744)
0620 20:28:02.719229 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.719323 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 20:28:02.720304 23083 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:39757: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
I20260620 20:28:02.771560 23083 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } attempt: 1
I20260620 20:28:02.772749 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.772292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:39757 (local address 127.0.0.1:51748)
0620 20:28:02.772501 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.772597 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 20:28:02.773901 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.773557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:39757 (local address 127.0.0.1:51756)
0620 20:28:02.773707 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.773780 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 20:28:02.774237 23083 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:39757: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
I20260620 20:28:02.789083 23157 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } attempt: 3
I20260620 20:28:02.790287 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.789818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:39757 (local address 127.0.0.1:51772)
0620 20:28:02.790030 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.790132 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 20:28:02.791427 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.791002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:39757 (local address 127.0.0.1:51774)
0620 20:28:02.791229 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.791304 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260620 20:28:02.791687 23157 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:39757: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
I20260620 20:28:02.842460 23083 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } attempt: 2
I20260620 20:28:02.843657 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.843132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:39757 (local address 127.0.0.1:51788)
0620 20:28:02.843396 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.843495 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 20:28:02.844725 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.844372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:39757 (local address 127.0.0.1:51802)
0620 20:28:02.844549 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.844626 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 20:28:02.844988 23083 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:39757: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
W20260620 20:28:02.913770 23164 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 20:28:02.914115 23164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 20:28:02.914247 23164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 20:28:02.914371 23164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 20:28:02.921792 23164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 20:28:02.921882 23164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 20:28:02.921967 23164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 20:28:02.922060 23164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 20:28:02.922165 23164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 20:28:02.932418 23164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.235.148:43849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.235.190:46263,127.31.235.189:39573,127.31.235.188:39757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/krb5kdc/kudu_127.31.235.188.keytab
--principal=kudu/127.31.235.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.235.188:39757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.235.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 18:55:48 UTC on ccycloud.quasar-mahdxv.root.comops.site
build id 4457677
ASAN enabled
I20260620 20:28:02.933777 23164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 20:28:02.934928 23157 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } attempt: 4
I20260620 20:28:02.935148 23164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 20:28:02.936096 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.935611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:39757 (local address 127.0.0.1:51816)
0620 20:28:02.935857 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.935937 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 20:28:02.937186 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.936779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:39757 (local address 127.0.0.1:51822)
0620 20:28:02.936951 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.937056 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 20:28:02.937398 23157 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:39757: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
W20260620 20:28:02.947559 23170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:28:02.947602 23172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 20:28:02.947785 23169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 20:28:02.948446 23164 server_base.cc:1034] running on GCE node
I20260620 20:28:02.949224 23164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 20:28:02.950723 23164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 20:28:02.951229 23083 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } attempt: 3
I20260620 20:28:02.951965 23164 hybrid_clock.cc:648] HybridClock initialized: now 1781987282951895 us; error 51 us; skew 500 ppm
I20260620 20:28:02.952292 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.951810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:39757 (local address 127.0.0.1:51836)
0620 20:28:02.952036 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.952133 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 20:28:02.952297 23164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 20:28:02.953308 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:02.952902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:39757 (local address 127.0.0.1:51852)
0620 20:28:02.953116 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:02.953197 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260620 20:28:02.953614 23083 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.235.188:39757: Network error: Client connection negotiation failed: client connection to 127.31.235.188:39757: connect: Connection refused (error 111)
Jun 20 20:28:02 dist-test-slave-zzwf krb5kdc[23003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987282, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 20:28:02.956424 23164 init.cc:377] Logged in from keytab as kudu/127.31.235.188@KRBTEST.COM (short username kudu)
I20260620 20:28:02.958313 23164 webserver.cc:460] Webserver started at http://127.31.235.188:43749/ using document root <none> and password file <none>
I20260620 20:28:02.959101 23164 fs_manager.cc:359] Metadata directory not provided
I20260620 20:28:02.959237 23164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 20:28:02.959612 23164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 20:28:02.963136 23164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-2/data/instance:
uuid: "3fa82b2ac36447438d6891ca39d1b4d0"
format_stamp: "Formatted at 2026-06-20 20:28:02 on dist-test-slave-zzwf"
server_key: "e1afc8fcee471b4c69891198ce1341f2"
server_key_iv: "36d53e4a1ac5623757b1553e171b8f99"
server_key_version: "encryptionkey@0"
I20260620 20:28:02.964046 23164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-2/wal/instance:
uuid: "3fa82b2ac36447438d6891ca39d1b4d0"
format_stamp: "Formatted at 2026-06-20 20:28:02 on dist-test-slave-zzwf"
server_key: "e1afc8fcee471b4c69891198ce1341f2"
server_key_iv: "36d53e4a1ac5623757b1553e171b8f99"
server_key_version: "encryptionkey@0"
I20260620 20:28:02.969396 23164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 20:28:02.973033 23179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 20:28:02.974493 23164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 20:28:02.974701 23164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-2/data,/tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-2/wal
uuid: "3fa82b2ac36447438d6891ca39d1b4d0"
format_stamp: "Formatted at 2026-06-20 20:28:02 on dist-test-slave-zzwf"
server_key: "e1afc8fcee471b4c69891198ce1341f2"
server_key_iv: "36d53e4a1ac5623757b1553e171b8f99"
server_key_version: "encryptionkey@0"
I20260620 20:28:02.974939 23164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 20:28:02.996510 23164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 20:28:03.000690 23164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 20:28:03.000978 23164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 20:28:03.019536 23164 rpc_server.cc:307] RPC server started. Bound to: 127.31.235.188:39757
I20260620 20:28:03.019572 23230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.235.188:39757 every 8 connection(s)
I20260620 20:28:03.020869 23164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-2/data/info.pb
I20260620 20:28:03.024621 23231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 20:28:03.028843 32686 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu as pid 23164
I20260620 20:28:03.029052 32686 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskgJsYo5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781986916608810-32686-0/minicluster-data/master-2/wal/instance
I20260620 20:28:03.029553 32686 external_mini_cluster.cc:1428] Setting key cb85e2d6c46d316643a33bb2e4396bd8
I20260620 20:28:03.032078 23231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46263 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:28:03 dist-test-slave-zzwf krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.188@KRBTEST.COM
Jun 20 20:28:03 dist-test-slave-zzwf krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987282, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.188@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:28:03.051081 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:03.036055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41718 (local address 127.31.235.190:46263)
0620 20:28:03.036278 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.036287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.036858 (+ 571us) server_negotiation.cc:407] Connection header received
0620 20:28:03.037091 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.037095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.037155 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.037263 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.038471 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.039128 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.040759 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.041373 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.041469 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:03.045610 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:28:03.045615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:28:03.045619 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:28:03.045651 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:28:03.048186 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.048888 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.048896 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.048903 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.049032 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.049440 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.049449 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.049451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.049764 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:28:03.049940 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 20:28:03.050472 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 20:28:03.050694 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 20:28:03.052040 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:03.031370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51864 (local address 127.31.235.188:39757)
0620 20:28:03.032125 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.032135 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.032179 (+ 44us) server_negotiation.cc:407] Connection header received
0620 20:28:03.034034 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.034056 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.034492 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.034793 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.036474 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.038092 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.039485 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.040116 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.040253 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:03.045145 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:28:03.045159 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:28:03.045175 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:28:03.045215 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:28:03.048296 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.048941 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.048950 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.048953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.049112 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.049483 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.049512 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.049516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.050007 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:28:03.050312 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 20:28:03.050610 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 20:28:03.051079 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":182,"server-negotiator.queue_time_us":555,"thread_start_us":360,"threads_started":1}
I20260620 20:28:03.052081 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:03.036030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.190:46263 (local address 127.0.0.1:41718)
0620 20:28:03.036725 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:03.036765 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 20:28:03.036949 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:28:03.037322 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:28:03.037329 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:28:03.038007 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:28:03.038322 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:28:03.038335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.039283 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.039290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:28:03.040547 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:28:03.040562 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.041612 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.041619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:28:03.041818 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:03.042445 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:28:03.042486 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:28:03.045309 (+ 2823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:28:03.048320 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:28:03.048332 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:28:03.048346 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:28:03.048751 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:28:03.049147 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:28:03.049152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:28:03.049156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:28:03.049309 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:28:03.049914 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:28:03.049922 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:28:03.050128 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 20:28:03.050385 (+ 257us) client_negotiation.cc:240] Negotiation successful
0620 20:28:03.050890 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":340,"threads_started":1}
I20260620 20:28:03.053699 23231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 20:28:03 dist-test-slave-zzwf krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987282, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.188@KRBTEST.COM for kudu/127.31.235.189@KRBTEST.COM
Jun 20 20:28:03 dist-test-slave-zzwf krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.190@KRBTEST.COM
I20260620 20:28:03.068547 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:03.054497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.189:39573 (local address 127.0.0.1:52418)
0620 20:28:03.054671 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:03.054689 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 20:28:03.054799 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:28:03.055157 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:28:03.055179 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:28:03.055651 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:28:03.055836 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:28:03.055847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.057306 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.057310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:28:03.058750 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:28:03.058760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.059679 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.059683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:28:03.059817 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:03.060390 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:28:03.060406 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:28:03.062672 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:28:03.066203 (+ 3531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:28:03.066210 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:28:03.066213 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:28:03.066576 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:28:03.066941 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:28:03.066946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:28:03.066950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:28:03.067087 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:28:03.067589 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:28:03.067593 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:28:03.067747 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 20:28:03.067894 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 20:28:03.068173 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 20:28:03.068574 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:03.054503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52418 (local address 127.31.235.189:39573)
0620 20:28:03.054685 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.054692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.054755 (+ 63us) server_negotiation.cc:407] Connection header received
0620 20:28:03.054931 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.054936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.055030 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.055135 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.056009 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.057144 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.058877 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.059546 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.059671 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:03.062894 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:28:03.062901 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:28:03.062907 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:28:03.062954 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:28:03.066075 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.066699 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.066710 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.066713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.066833 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.067193 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.067199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.067203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.067469 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:28:03.067629 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 20:28:03.067993 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 20:28:03.068198 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 20:28:03.069574 23231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } has no permanent_uuid. Determining permanent_uuid...
I20260620 20:28:03.070353 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:03.055540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41722 (local address 127.31.235.190:46263)
0620 20:28:03.055697 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.055703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.055720 (+ 17us) server_negotiation.cc:407] Connection header received
0620 20:28:03.055801 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.055807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.055883 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.055980 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.057414 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.058535 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.060155 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.060587 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.060735 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:03.065066 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:28:03.065071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:28:03.065074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:28:03.065105 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:28:03.067662 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.068287 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.068294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.068298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.068457 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.068898 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.068903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.068906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.069200 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:28:03.069518 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 20:28:03.069832 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 20:28:03.070046 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 20:28:03 dist-test-slave-zzwf krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987282, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.188@KRBTEST.COM for kudu/127.31.235.188@KRBTEST.COM
Jun 20 20:28:03 dist-test-slave-zzwf krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.235.189@KRBTEST.COM
I20260620 20:28:03.083630 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:03.070367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:39757 (local address 127.0.0.1:51878)
0620 20:28:03.070567 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:03.070586 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 20:28:03.070706 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:28:03.071119 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:28:03.071123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:28:03.071586 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:28:03.071712 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:28:03.071742 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.073085 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.073090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:28:03.074137 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:28:03.074150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.074901 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.074907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:28:03.075173 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:03.075796 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:28:03.075813 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:28:03.078080 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:28:03.081148 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:28:03.081156 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:28:03.081159 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:28:03.081561 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:28:03.082011 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:28:03.082014 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:28:03.082017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:28:03.082160 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:28:03.082830 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:28:03.082839 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:28:03.082992 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 20:28:03.083126 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 20:28:03.083336 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 20:28:03.083935 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:03.070756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.31.235.188:39757)
0620 20:28:03.070901 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.070905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.070919 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:28:03.070954 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.070957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.071006 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.071107 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.071901 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.072955 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.074300 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.074751 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.074894 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:03.078318 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:28:03.078323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:28:03.078326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:28:03.078352 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:28:03.081017 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.081681 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.081685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.081688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.081861 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.082293 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.082300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.082304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.082669 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:28:03.082826 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 20:28:03.083286 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 20:28:03.083537 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 20:28:03.087129 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:03.072066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.31.235.189:39573)
0620 20:28:03.072232 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.072237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.072251 (+ 14us) server_negotiation.cc:407] Connection header received
0620 20:28:03.072448 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.072453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.072536 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.072630 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.073985 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.074736 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.076065 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.076487 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.076621 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:03.081033 (+ 4412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:28:03.081037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:28:03.081040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:28:03.081070 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:28:03.084312 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.085144 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.085148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.085150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.085306 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.085783 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.085794 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.085796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.086050 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:28:03.086326 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 20:28:03.086603 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 20:28:03.086838 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 20:28:03.089275 23231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0: Bootstrap starting.
I20260620 20:28:03.091923 32686 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 20:28:03.092870 23231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0: Neither blocks nor log segments found. Creating new log.
I20260620 20:28:03.094060 23231 log.cc:824] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0: Log is configured to *not* fsync() on all Append() calls
I20260620 20:28:03.096792 23231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0: No bootstrap required, opened a new log
I20260620 20:28:03.094290 23058 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2571456d35854c32b00534564be74ab0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41722
I20260620 20:28:03.103880 23231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571456d35854c32b00534564be74ab0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46263 } } peers { permanent_uuid: "95aea5a6bc5441d8aa61deacfe11141d" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } } peers { permanent_uuid: "3fa82b2ac36447438d6891ca39d1b4d0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } }
I20260620 20:28:03.104212 23231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fa82b2ac36447438d6891ca39d1b4d0, State: Initialized, Role: FOLLOWER
I20260620 20:28:03.104699 23231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571456d35854c32b00534564be74ab0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46263 } } peers { permanent_uuid: "95aea5a6bc5441d8aa61deacfe11141d" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } } peers { permanent_uuid: "3fa82b2ac36447438d6891ca39d1b4d0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } }
I20260620 20:28:03.105814 23237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571456d35854c32b00534564be74ab0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46263 } } peers { permanent_uuid: "95aea5a6bc5441d8aa61deacfe11141d" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } } peers { permanent_uuid: "3fa82b2ac36447438d6891ca39d1b4d0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } } }
I20260620 20:28:03.106220 23237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:28:03.106797 23231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 20:28:03.117039 23248 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 20:28:03.117180 23248 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 20:28:03.114909 23164 master_runner.cc:186] Error getting master registration for 127.31.235.190:46263: OK, instance_id { permanent_uuid: "2571456d35854c32b00534564be74ab0" instance_seqno: 1781987281979504 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:28:03.126470 23164 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:28:03.130865 23083 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } attempt: 4
Jun 20 20:28:03 dist-test-slave-zzwf krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987281, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.190@KRBTEST.COM for kudu/127.31.235.188@KRBTEST.COM
I20260620 20:28:03.147938 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:03.131567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:39757 (local address 127.0.0.1:51882)
0620 20:28:03.131859 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:03.131881 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 20:28:03.132004 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:28:03.132399 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:28:03.132406 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:28:03.132948 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:28:03.133192 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:28:03.133205 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.134721 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.134727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:28:03.136214 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:28:03.136231 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.137264 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.137271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:28:03.137537 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:03.138210 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:28:03.138234 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:28:03.141126 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:28:03.145142 (+ 4016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:28:03.145147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:28:03.145152 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:28:03.145689 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:28:03.146088 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:28:03.146093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:28:03.146097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:28:03.146255 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:28:03.146925 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:28:03.146931 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:28:03.147094 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 20:28:03.147241 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 20:28:03.147518 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260620 20:28:03.147975 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:03.131703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51882 (local address 127.31.235.188:39757)
0620 20:28:03.131925 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.131931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.131957 (+ 26us) server_negotiation.cc:407] Connection header received
0620 20:28:03.132135 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.132142 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.132225 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.132334 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.133341 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.134561 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.136358 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.137104 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.137226 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:03.141330 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:28:03.141337 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:28:03.141343 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:28:03.141390 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:28:03.144996 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.145810 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.145816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.145820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.145965 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.146364 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.146376 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.146380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.146794 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:28:03.146978 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 20:28:03.147376 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 20:28:03.147606 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 20:28:03.153972 23083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0: Bootstrap starting.
I20260620 20:28:03.158073 23083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0: Neither blocks nor log segments found. Creating new log.
I20260620 20:28:03.159191 23083 log.cc:824] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0: Log is configured to *not* fsync() on all Append() calls
I20260620 20:28:03.162074 23083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0: No bootstrap required, opened a new log
I20260620 20:28:03.169735 23083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571456d35854c32b00534564be74ab0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46263 } } peers { permanent_uuid: "95aea5a6bc5441d8aa61deacfe11141d" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } } peers { permanent_uuid: "3fa82b2ac36447438d6891ca39d1b4d0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } }
I20260620 20:28:03.170145 23083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2571456d35854c32b00534564be74ab0, State: Initialized, Role: FOLLOWER
I20260620 20:28:03.170645 23083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571456d35854c32b00534564be74ab0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46263 } } peers { permanent_uuid: "95aea5a6bc5441d8aa61deacfe11141d" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } } peers { permanent_uuid: "3fa82b2ac36447438d6891ca39d1b4d0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } }
I20260620 20:28:03.171713 23251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571456d35854c32b00534564be74ab0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46263 } } peers { permanent_uuid: "95aea5a6bc5441d8aa61deacfe11141d" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } } peers { permanent_uuid: "3fa82b2ac36447438d6891ca39d1b4d0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } } }
I20260620 20:28:03.172066 23251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:28:03.172835 23083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:28:03.172983 23058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 20:28:03.173213 23058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:28:03.176990 23058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571456d35854c32b00534564be74ab0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46263 } } peers { permanent_uuid: "95aea5a6bc5441d8aa61deacfe11141d" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } } peers { permanent_uuid: "3fa82b2ac36447438d6891ca39d1b4d0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } }
I20260620 20:28:03.179853 23058 leader_election.cc:288] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [CANDIDATE]: Term 1 election: Requested vote from peers 95aea5a6bc5441d8aa61deacfe11141d (127.31.235.189:39573), 3fa82b2ac36447438d6891ca39d1b4d0 (127.31.235.188:39757)
I20260620 20:28:03.179584 23206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2571456d35854c32b00534564be74ab0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3fa82b2ac36447438d6891ca39d1b4d0"
I20260620 20:28:03.180320 23206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:28:03.185168 23206 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2571456d35854c32b00534564be74ab0 in term 1.
I20260620 20:28:03.189615 23033 leader_election.cc:302] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2571456d35854c32b00534564be74ab0, 3fa82b2ac36447438d6891ca39d1b4d0; no voters:
I20260620 20:28:03.190696 23251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 20:28:03.204382 23251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [term 1 LEADER]: Becoming Leader. State: Replica: 2571456d35854c32b00534564be74ab0, State: Running, Role: LEADER
I20260620 20:28:03.206018 23251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571456d35854c32b00534564be74ab0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46263 } } peers { permanent_uuid: "95aea5a6bc5441d8aa61deacfe11141d" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } } peers { permanent_uuid: "3fa82b2ac36447438d6891ca39d1b4d0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } }
W20260620 20:28:03.218180 23037 tablet.cc:2369] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:28:03.220463 23255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2571456d35854c32b00534564be74ab0. Latest consensus state: current_term: 1 leader_uuid: "2571456d35854c32b00534564be74ab0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571456d35854c32b00534564be74ab0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46263 } } peers { permanent_uuid: "95aea5a6bc5441d8aa61deacfe11141d" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } } peers { permanent_uuid: "3fa82b2ac36447438d6891ca39d1b4d0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } } }
I20260620 20:28:03.221220 23255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:28:03.230147 23266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 20:28:03.236627 23157 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } attempt: 5
W20260620 20:28:03.253221 23185 tablet.cc:2369] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 20:28:03.254784 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:03.212081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.31.235.190:46263)
0620 20:28:03.212270 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.212278 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.216265 (+ 3987us) server_negotiation.cc:407] Connection header received
0620 20:28:03.217345 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.217352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.217451 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.217551 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.222942 (+ 5391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.224348 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.232066 (+ 7718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.232868 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.232986 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:03.241190 (+ 8204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:28:03.241197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:28:03.241201 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:28:03.241260 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:28:03.245036 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.249927 (+ 4891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.249939 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.249943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.252310 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.252823 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.252831 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.252836 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.253209 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:28:03.253460 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 20:28:03.254080 (+ 620us) server_negotiation.cc:299] Negotiation successful
0620 20:28:03.254350 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 20:28:03.248687 23266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 20:28:03.260179 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:03.213760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52426 (local address 127.31.235.189:39573)
0620 20:28:03.213957 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.213963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.224605 (+ 10642us) server_negotiation.cc:407] Connection header received
0620 20:28:03.225285 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.225292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.225389 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.225468 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.230025 (+ 4557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.231356 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.235456 (+ 4100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.236216 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.236330 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:03.245443 (+ 9113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:28:03.245450 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:28:03.245456 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:28:03.245525 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:28:03.251130 (+ 5605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.254380 (+ 3250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.254390 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.254396 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.254629 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.255210 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.255216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.255220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.257142 (+ 1922us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:28:03.257455 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 20:28:03.259499 (+ 2044us) server_negotiation.cc:299] Negotiation successful
0620 20:28:03.259776 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 20:28:03.271716 23206 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0 [term 1 FOLLOWER]: Refusing update from remote peer 2571456d35854c32b00534564be74ab0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
Jun 20 20:28:03 dist-test-slave-zzwf krb5kdc[23003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781987282, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.235.189@KRBTEST.COM for kudu/127.31.235.188@KRBTEST.COM
I20260620 20:28:03.275188 23251 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fa82b2ac36447438d6891ca39d1b4d0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:28:03.275820 23234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:03.217643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51888 (local address 127.31.235.188:39757)
0620 20:28:03.217836 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.217843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.242747 (+ 24904us) server_negotiation.cc:407] Connection header received
0620 20:28:03.243063 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.243073 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.243191 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.243306 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.246346 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.247621 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.262895 (+ 15274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.263742 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.263863 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:03.267676 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:28:03.267683 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:28:03.267688 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:28:03.267729 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:28:03.272058 (+ 4329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.273092 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.273100 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.273105 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.273288 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.273825 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.273834 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.273837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.274204 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:28:03.274499 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 20:28:03.275137 (+ 638us) server_negotiation.cc:299] Negotiation successful
0620 20:28:03.275408 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 20:28:03.286782 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:03.237257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.235.188:39757 (local address 127.0.0.1:51898)
0620 20:28:03.237775 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 20:28:03.237802 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 20:28:03.237933 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 20:28:03.262974 (+ 25041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 20:28:03.262980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 20:28:03.263578 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0620 20:28:03.263827 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:28:03.263841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.265791 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.265798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:28:03.267445 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 20:28:03.267461 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.268568 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.268576 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 20:28:03.268906 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:03.272387 (+ 3481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 20:28:03.272414 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 20:28:03.275661 (+ 3247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 20:28:03.283556 (+ 7895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:28:03.283562 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:28:03.283567 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:28:03.284104 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:28:03.284516 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 20:28:03.284522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 20:28:03.284525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 20:28:03.284731 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 20:28:03.285648 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 20:28:03.285654 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 20:28:03.285831 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 20:28:03.285989 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 20:28:03.286339 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338}
I20260620 20:28:03.288022 23269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:03.238168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51898 (local address 127.31.235.188:39757)
0620 20:28:03.262493 (+ 24325us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.262500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.262525 (+ 25us) server_negotiation.cc:407] Connection header received
0620 20:28:03.262583 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.262588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.262674 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.263977 (+ 1303us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.264224 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.265594 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.267639 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:03.268381 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:03.275826 (+ 7445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:03.276099 (+ 273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 20:28:03.276105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 20:28:03.276110 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 20:28:03.276143 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 20:28:03.283350 (+ 7207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.284248 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.284252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.284254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.284379 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 20:28:03.284881 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 20:28:03.284887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 20:28:03.284891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 20:28:03.285439 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 20:28:03.287176 (+ 1737us) server_negotiation.cc:1035] Waiting for connection context
0620 20:28:03.287316 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 20:28:03.287552 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24160,"thread_start_us":24600,"threads_started":1}
I20260620 20:28:03.276332 23016 master_runner.cc:186] Error getting master registration for 127.31.235.189:39573: OK, instance_id { permanent_uuid: "95aea5a6bc5441d8aa61deacfe11141d" instance_seqno: 1781987282498546 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 20:28:03.289420 23016 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 20:28:03.294447 23157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95aea5a6bc5441d8aa61deacfe11141d: Bootstrap starting.
I20260620 20:28:03.296078 23237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2571456d35854c32b00534564be74ab0. Latest consensus state: current_term: 1 leader_uuid: "2571456d35854c32b00534564be74ab0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571456d35854c32b00534564be74ab0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46263 } } peers { permanent_uuid: "95aea5a6bc5441d8aa61deacfe11141d" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } } peers { permanent_uuid: "3fa82b2ac36447438d6891ca39d1b4d0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } } }
I20260620 20:28:03.296409 23237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:28:03.298991 23157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95aea5a6bc5441d8aa61deacfe11141d: Neither blocks nor log segments found. Creating new log.
I20260620 20:28:03.300395 23157 log.cc:824] T 00000000000000000000000000000000 P 95aea5a6bc5441d8aa61deacfe11141d: Log is configured to *not* fsync() on all Append() calls
I20260620 20:28:03.301065 23273 mvcc.cc:205] Tried to move back new op lower bound from 7299019912261308416 to 7299019912030154752. Current Snapshot: MvccSnapshot[applied={T|T < 7299019912261308416}]
I20260620 20:28:03.302150 23237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2571456d35854c32b00534564be74ab0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571456d35854c32b00534564be74ab0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46263 } } peers { permanent_uuid: "95aea5a6bc5441d8aa61deacfe11141d" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } } peers { permanent_uuid: "3fa82b2ac36447438d6891ca39d1b4d0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } } }
I20260620 20:28:03.302420 23237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:28:03.313911 23255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2571456d35854c32b00534564be74ab0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571456d35854c32b00534564be74ab0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46263 } } peers { permanent_uuid: "95aea5a6bc5441d8aa61deacfe11141d" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } } peers { permanent_uuid: "3fa82b2ac36447438d6891ca39d1b4d0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } } }
I20260620 20:28:03.314213 23255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [sys.catalog]: This master's current role is: LEADER
I20260620 20:28:03.315155 23266 catalog_manager.cc:1347] Generated new cluster ID: 394bc3a1b2814f9eb10fcb8ad61a3696
I20260620 20:28:03.315289 23266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 20:28:03.322000 23157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95aea5a6bc5441d8aa61deacfe11141d: No bootstrap required, opened a new log
I20260620 20:28:03.333251 23266 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 20:28:03.337441 23266 catalog_manager.cc:1504] Loading token signing keys...
I20260620 20:28:03.342516 23157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95aea5a6bc5441d8aa61deacfe11141d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571456d35854c32b00534564be74ab0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46263 } } peers { permanent_uuid: "95aea5a6bc5441d8aa61deacfe11141d" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } } peers { permanent_uuid: "3fa82b2ac36447438d6891ca39d1b4d0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } }
I20260620 20:28:03.342998 23157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95aea5a6bc5441d8aa61deacfe11141d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95aea5a6bc5441d8aa61deacfe11141d, State: Initialized, Role: FOLLOWER
I20260620 20:28:03.343658 23157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95aea5a6bc5441d8aa61deacfe11141d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571456d35854c32b00534564be74ab0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46263 } } peers { permanent_uuid: "95aea5a6bc5441d8aa61deacfe11141d" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } } peers { permanent_uuid: "3fa82b2ac36447438d6891ca39d1b4d0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } }
I20260620 20:28:03.350144 23266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0: Generated new TSK 0
I20260620 20:28:03.351791 23266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 20:28:03.354161 23281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95aea5a6bc5441d8aa61deacfe11141d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571456d35854c32b00534564be74ab0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46263 } } peers { permanent_uuid: "95aea5a6bc5441d8aa61deacfe11141d" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } } peers { permanent_uuid: "3fa82b2ac36447438d6891ca39d1b4d0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } } }
I20260620 20:28:03.354722 23281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95aea5a6bc5441d8aa61deacfe11141d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:28:03.359513 23157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95aea5a6bc5441d8aa61deacfe11141d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 20:28:03.360097 23131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95aea5a6bc5441d8aa61deacfe11141d [term 0 FOLLOWER]: Advancing to term 1
I20260620 20:28:03.364202 23131 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 95aea5a6bc5441d8aa61deacfe11141d [term 1 FOLLOWER]: Refusing update from remote peer 2571456d35854c32b00534564be74ab0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 20:28:03.365134 23251 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95aea5a6bc5441d8aa61deacfe11141d" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 20:28:03.369686 23132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2571456d35854c32b00534564be74ab0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "95aea5a6bc5441d8aa61deacfe11141d"
I20260620 20:28:03.419466 23281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95aea5a6bc5441d8aa61deacfe11141d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2571456d35854c32b00534564be74ab0. Latest consensus state: current_term: 1 leader_uuid: "2571456d35854c32b00534564be74ab0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571456d35854c32b00534564be74ab0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46263 } } peers { permanent_uuid: "95aea5a6bc5441d8aa61deacfe11141d" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } } peers { permanent_uuid: "3fa82b2ac36447438d6891ca39d1b4d0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } } }
I20260620 20:28:03.419823 23281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95aea5a6bc5441d8aa61deacfe11141d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:28:03.420235 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95aea5a6bc5441d8aa61deacfe11141d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2571456d35854c32b00534564be74ab0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571456d35854c32b00534564be74ab0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46263 } } peers { permanent_uuid: "95aea5a6bc5441d8aa61deacfe11141d" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } } peers { permanent_uuid: "3fa82b2ac36447438d6891ca39d1b4d0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } } }
I20260620 20:28:03.420921 23132 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 95aea5a6bc5441d8aa61deacfe11141d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2571456d35854c32b00534564be74ab0 for term 1 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260620 20:28:03.422976 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95aea5a6bc5441d8aa61deacfe11141d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 20:28:03.443667 23300 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260620 20:28:03.488582 23269 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:28:03.480218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51904 (local address 127.31.235.188:39757)
0620 20:28:03.480407 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.480413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.486122 (+ 5709us) server_negotiation.cc:407] Connection header received
0620 20:28:03.486474 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.486484 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.486637 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.486732 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.488221 (+ 1489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51904: BlockingRecv error: recv got EOF from 127.0.0.1:51904 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 20:28:03.495752 23087 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:28:03.479199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41744 (local address 127.31.235.190:46263)
0620 20:28:03.479485 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.479492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.493571 (+ 14079us) server_negotiation.cc:407] Connection header received
0620 20:28:03.494162 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.494170 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.494286 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.494360 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.495471 (+ 1111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41744: BlockingRecv error: recv got EOF from 127.0.0.1:41744 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
W20260620 20:28:03.497610 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:28:03.479803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52440 (local address 127.31.235.189:39573)
0620 20:28:03.480000 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.480007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.489327 (+ 9320us) server_negotiation.cc:407] Connection header received
0620 20:28:03.489537 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.489543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.489656 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.489783 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.497313 (+ 7530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52440: BlockingRecv error: recv got EOF from 127.0.0.1:52440 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 20:28:03.503538 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:28:03.501527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.31.235.189:39573)
0620 20:28:03.501711 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.501719 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.502006 (+ 287us) server_negotiation.cc:407] Connection header received
0620 20:28:03.502217 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.502226 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.502375 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.502501 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.503271 (+ 770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52454: BlockingRecv error: recv got EOF from 127.0.0.1:52454 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 20:28:03.541124 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:28:03.539079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52460 (local address 127.31.235.189:39573)
0620 20:28:03.539306 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.539316 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.539537 (+ 221us) server_negotiation.cc:407] Connection header received
0620 20:28:03.539735 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.539741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.539857 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.539980 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.540877 (+ 897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52460: BlockingRecv error: recv got EOF from 127.0.0.1:52460 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260620 20:28:03.551538 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:28:03.547349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.31.235.189:39573)
0620 20:28:03.547526 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.547532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.550079 (+ 2547us) server_negotiation.cc:407] Connection header received
0620 20:28:03.550230 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.550236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.550340 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.550461 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.551277 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52462: BlockingRecv error: recv got EOF from 127.0.0.1:52462 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 20:28:03.563642 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:28:03.561290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52472 (local address 127.31.235.189:39573)
0620 20:28:03.561522 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.561528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.562055 (+ 527us) server_negotiation.cc:407] Connection header received
0620 20:28:03.562227 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.562232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.562352 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.562468 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.563387 (+ 919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52472: BlockingRecv error: recv got EOF from 127.0.0.1:52472 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260620 20:28:03.577792 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:28:03.575421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52474 (local address 127.31.235.189:39573)
0620 20:28:03.575621 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.575627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.576157 (+ 530us) server_negotiation.cc:407] Connection header received
0620 20:28:03.576372 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.576379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.576490 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.576617 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.577480 (+ 863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52474: BlockingRecv error: recv got EOF from 127.0.0.1:52474 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 20:28:03.600063 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:28:03.597624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52490 (local address 127.31.235.189:39573)
0620 20:28:03.597846 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.597853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.598388 (+ 535us) server_negotiation.cc:407] Connection header received
0620 20:28:03.598590 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.598597 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.598707 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.598847 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.599784 (+ 937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52490: BlockingRecv error: recv got EOF from 127.0.0.1:52490 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 20:28:03.638316 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:28:03.635891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.31.235.189:39573)
0620 20:28:03.636084 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.636093 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.636659 (+ 566us) server_negotiation.cc:407] Connection header received
0620 20:28:03.636878 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.636885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.636993 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.637125 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.638010 (+ 885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52498: BlockingRecv error: recv got EOF from 127.0.0.1:52498 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 20:28:03.710929 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:28:03.708205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52502 (local address 127.31.235.189:39573)
0620 20:28:03.708412 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.708421 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.708981 (+ 560us) server_negotiation.cc:407] Connection header received
0620 20:28:03.709312 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.709319 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.709414 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.709521 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.710667 (+ 1146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52502: BlockingRecv error: recv got EOF from 127.0.0.1:52502 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 20:28:03.844367 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:28:03.842308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52504 (local address 127.31.235.189:39573)
0620 20:28:03.842534 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:03.842542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:03.842832 (+ 290us) server_negotiation.cc:407] Connection header received
0620 20:28:03.843056 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:03.843062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:03.843146 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:03.843252 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:03.844091 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52504: BlockingRecv error: recv got EOF from 127.0.0.1:52504 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:28:03.856439 23251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2571456d35854c32b00534564be74ab0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2571456d35854c32b00534564be74ab0" member_type: VOTER last_known_addr { host: "127.31.235.190" port: 46263 } } peers { permanent_uuid: "95aea5a6bc5441d8aa61deacfe11141d" member_type: VOTER last_known_addr { host: "127.31.235.189" port: 39573 } } peers { permanent_uuid: "3fa82b2ac36447438d6891ca39d1b4d0" member_type: VOTER last_known_addr { host: "127.31.235.188" port: 39757 } } }
I20260620 20:28:03.856842 23251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2571456d35854c32b00534564be74ab0 [sys.catalog]: This master's current role is: LEADER
W20260620 20:28:04.105993 23162 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 20:28:04.104070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52516 (local address 127.31.235.189:39573)
0620 20:28:04.104270 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:04.104275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:04.104525 (+ 250us) server_negotiation.cc:407] Connection header received
0620 20:28:04.104663 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:04.104667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:04.104742 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:04.104841 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 20:28:04.105732 (+ 891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52516: BlockingRecv error: recv got EOF from 127.0.0.1:52516 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 20:28:04.119150 23248 catalog_manager.cc:1259] Loaded cluster ID: 394bc3a1b2814f9eb10fcb8ad61a3696
I20260620 20:28:04.119256 23248 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0: loading cluster ID for follower catalog manager: success
I20260620 20:28:04.121758 23248 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0: acquiring CA information for follower catalog manager: success
I20260620 20:28:04.123600 23248 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3fa82b2ac36447438d6891ca39d1b4d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:28:04.448153 23300 catalog_manager.cc:1259] Loaded cluster ID: 394bc3a1b2814f9eb10fcb8ad61a3696
I20260620 20:28:04.448310 23300 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 95aea5a6bc5441d8aa61deacfe11141d: loading cluster ID for follower catalog manager: success
I20260620 20:28:04.450951 23300 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 95aea5a6bc5441d8aa61deacfe11141d: acquiring CA information for follower catalog manager: success
I20260620 20:28:04.452749 23300 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 95aea5a6bc5441d8aa61deacfe11141d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 20:28:04.628988 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:04.621974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.31.235.189:39573)
0620 20:28:04.622710 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:04.622717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:04.622947 (+ 230us) server_negotiation.cc:407] Connection header received
0620 20:28:04.623129 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:04.623133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:04.623279 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:04.623413 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:28:04.623863 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:04.624663 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:04.626501 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:04.626965 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:04.627073 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:04.627675 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:28:04.628267 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:28:04.628384 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 20:28:04.628539 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 20:28:04.628611 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":402,"threads_started":1}
I20260620 20:28:04.642079 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:04.635201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51916 (local address 127.31.235.188:39757)
0620 20:28:04.635846 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:04.635853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:04.635876 (+ 23us) server_negotiation.cc:407] Connection header received
0620 20:28:04.635988 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:04.635994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:04.636193 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:04.636376 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:28:04.636732 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:04.637905 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:04.639201 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:04.639866 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:04.640000 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:04.640535 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:28:04.641252 (+ 717us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:28:04.641435 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 20:28:04.641657 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 20:28:04.641744 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":380,"threads_started":1}
I20260620 20:28:04.670810 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:04.648166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41754 (local address 127.31.235.190:46263)
0620 20:28:04.661097 (+ 12931us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:04.661106 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:04.661813 (+ 707us) server_negotiation.cc:407] Connection header received
0620 20:28:04.662095 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:04.662104 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:04.662524 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:04.662740 (+ 216us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:28:04.663837 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:04.665231 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:04.668034 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:04.668661 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:04.668754 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:04.669598 (+ 844us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:28:04.670059 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:28:04.670136 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 20:28:04.670356 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 20:28:04.670454 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12775,"thread_start_us":13811,"threads_started":1}
I20260620 20:28:04.671151 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:04.648345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52538 (local address 127.31.235.189:39573)
0620 20:28:04.648596 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:04.648604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:04.664226 (+ 15622us) server_negotiation.cc:407] Connection header received
0620 20:28:04.664499 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:04.664505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:04.664610 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:04.664673 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:28:04.665433 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:04.666652 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:04.668950 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:04.669639 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:04.669738 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:04.670365 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:28:04.670493 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:28:04.670550 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 20:28:04.670783 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 20:28:04.670865 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 20:28:04.674693 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 20:28:04.650781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51926 (local address 127.31.235.188:39757)
0620 20:28:04.651015 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 20:28:04.651024 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 20:28:04.668087 (+ 17063us) server_negotiation.cc:407] Connection header received
0620 20:28:04.668512 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 20:28:04.668518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 20:28:04.668616 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 20:28:04.668682 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 20:28:04.669468 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:04.670524 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:04.672499 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 20:28:04.673068 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 20:28:04.673164 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 20:28:04.673847 (+ 683us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 20:28:04.674031 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 20:28:04.674123 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 20:28:04.674279 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 20:28:04.674372 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 20:28:04.676582 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 23016
I20260620 20:28:04.677610 23078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:28:04.789328 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 23016
I20260620 20:28:04.818032 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 23090
I20260620 20:28:04.819156 23152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:28:04.925385 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 23090
I20260620 20:28:04.945401 32686 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu pid 23164
I20260620 20:28:04.946590 23226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 20:28:05.059295 32686 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskgJsYo5/build/asan/bin/kudu with pid 23164
2026-06-20T20:28:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4910 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9808 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (368416 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST